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An Analysis of the Detection of Repeated Signals 
in Noise by Binary Integration’ 


J. V. HARRINGTONT 


Summary—An analysis of the detection of repetitive signals in 
noise by binary integration techniques is made. An expression for 
the effective signal-to-noise ratio of the quantized video is obtained 
and is shown to apply to any half-wave second detector. A comparison 
of analog and digital integration is made, and it is further shown that 
digital integration is, at most, 1.9 db poorer due to the quantization 
loss. However, the loss due to nonideal analog integration can make 
the two types equivalent. The optimum settings for quantizer and 
counter thresholds are derived, and expressions for the final-detec- 
tion and false-alarm probabilities are determined. Lastly, the results 
are modified to include the effect of nonuniform amplitudes in the 
set of signals being quantized and integrated. 


INTRODUCTION 


ECENTLY, § signal-integration techniques have 
come into use as a means of improving the 
detection of repetitive signals in noise.” How- 

ever, the methods used to date for obtaining the desired 
signal storage, which is a necessary part of the integration 
process, have been primarily analog in nature. For 
example, delay-line integrators and storage-tube inte- 
grators have been developed which integrate by re- 
membering the waveform of the signal in noise and then 
superimposing successive samples to obtain the desired 
improvement. The theory is that, when successive repe- 
tition intervals of any radar video are added, the noise 
voltage increases roughly as the square root of the number 
of additions, and the signal increases as the number of 
additions. Hence the relative signal-to-noise ratio should 
increase as the square root of the number of samples 
added. This has been borne out by experiment and has 
been well reported in the literature. There are, however, 
several disadvantages to analog integrators: the rather 
large number of memory elements required to store the 
waveform of the signal, and the relatively short memory 
times; thus, in the radar case, an addition of some ten or 
twenty signals is regarded as a practical maximum. 

In order to obtain a system that will require fewer 
memory elements and that will remember signals over a 
preater number of samples, another method of integration 
in which quantized signals are added has come into use. 
While it is possible to quantize the signal amplitude into 
many discrete levels, the particular type of quantized 
signal integration to be discussed in this report is one in 
which the signals are quantized into two amplitude levels 
and, of course, are quantized in time between fixed time 


*The research in this document was supported jointly by the 
a Navy, and Air Force under contract with the Massachusetts 


nstitute of Technology, Cambridge, Mass. 

+ Staff Member, Massachusetts Institute of Technology. 

1 J. V. Harrington and T. F. Rogers, ‘‘Signal-to-noise improvement 
hrough integration in a storage tube,” Proc. I.R.E., vol. 38, pp. 
197-1203; October, 1950. (See also references contained therein.) 


markers. In the process of quantizing, if the complex 
signal and noise waveform between given time markers 
exceeds a predetermined amplitude, a standard pulse is 
generated at the end of the interval; if the threshold is 
not exceeded, no pulse is generated. The probability of 
obtaining a standard pulse can then be determined from 
the probability distribution function for the complex 
waveform in question. 

For example, the signal plus noise prior to quantizing 
might have the probability density shown in Fig. 1. If 
the threshold is set as indicated and the cumulative 
probability calculated, the new distribution function that 
characterizes the quantized video consists of two lines of 
magnitude P at 1 (pulse present) and magnitude 1 — P 
at 0 (pulse absent). 


Fig. 1—Signal probability densities before and after quantization. 


The method of integration then becomes a process of 
adding or counting a set of standard pulses with certain 
occurrence probabilities, instead of a process of super- 
imposing and adding signal waveforms in successive 
repetition intervals. Some other obvious advantages 
result from the employment of quantized video, as in 
computer use, switching and memory circuits composed 
solely of bistable elements can be used to sort and count 
these pulses. Somewhat greater reliability is potentially 
attainable over the analog integration techniques, and, 
with the required circuitry reduced to an array of bistable 
elements, the use of transistors becomes quite attractive. 

One of the first questions asked is that of the relative 
efficiency of binary integration as compared to analog 
integration. It is the purpose of this report to analyze the 
performance of the binary integration scheme, to indicate 
the threshold settings for best signal detection, and to 
compare the process with analog integration, indicating 
the relative merits and limitations of each method. 


Basis or ANALYSIS 


A simplified block diagram of a general binary inte- 
gration system is shown in Fig. 2 (next page). Signals 
coming from the second detector of the radar receiver must 
pass first through a threshold detector which quantizes the 
signal in amplitude and then through a quantizer which 
performs the same function in time. The quantized video 


2 IRE TRANSACTIONS 


is then gated in range so that pulses occurring in a given 
range interval are switched into a binary counter where 
they are counted over the number of repetition intervals 
in which the signal is expected to repeat. At the end of 
this period, the binary counter is sampled, and, if the 
number in the counter is some specified fraction of the 
total number of samples taken, a signal is judged to be 
the 
operation of this integrator arise: first, how far down into 


Immediately, two concerning 


present. j questions 
the noise level should the input threshold be set in order 
to give good detection probabilities on weak signals 
without flooding the integrator with signals due to noise; 
second, what should be the minimum count in the counter 
over a given number of samples in order that the prob- 
ability of detecting a signal, and not a noise pulse, shall 
be reasonably high. 


QUANTIZED 
VIDEO 
| TARGET 
RADAR | VIOEO|THRESHOLD QUANTIZER RANGE BINARY count | PULSE 
RECEIVER DETECTOR GATE COUNTER SAMPLER 
INPUT OUTPUT 
THRESHOLD THRESHOLD 


Fig. 2—Block diagram of binary integrator. 


These two questions cannot be answered independently, 
and it is desired to determine the best setting for both 
input and output thresholds for best detection of small 
signals. The procedure used in the analysis is straight- 
forward, and consists of using the known distribution 
functions for signal and noise in the output of the second 
detector in order to determine the occurrence probability 
for the quantized video pulse. From these, the discrete 
density functions for the sum of a given number of 
samples can be determined; and, finally, from these the 
detection probabilities following binary integration can 
be determined for noise alone and for signal plus noise. 
This procedure is followed for given input signal-to-noise 
ratios, for given settings of the input and output thresh- 
olds, and for integration of a given number of samples. 

In order to keep the analysis simple and yet to obtain 
some useful results, it will be assumed that the pulses 
being counted will have uniform probabilities of occur- 
rence. This immediately allows the use of the well-known 
binomial distribution and produces, in a straightforward 
fashion, a fairly useful result. It is implied in this assump- 
tion that all the samples being added have the same 
signal-to-noise ratio out of the second detector; this, in 
turn, implies that, in the radar case, the antenna is 
stationary. While this assumption of uniform occurrence 
probability is certainly not justified for a scanning radar, 
it does give a first approximation to the actual envelope 
of the signals, and the results can be subsequently modified 
to indicate the effect of a scanning antenna. 

At this point, in order to get some physical feeling for 
the problem being analyzed, it will probably be helpful 
to refer to the set of quantized radar signals shown in Fig. 
3. The picture shows successive repetition intervals of 
quantized radar video in the first 32 miles of range; here, 
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the dots represent detected signals in a given half-mile 
range interval. The total spread of the picture is of the 
order of 3 degrees, or several beamwidths of the radar 
used, and a sequence of dots at constant range represents 
the occurrence of the target. The densely occupied region 


Fig. 3—Quantized video pattern taken over 32 miles in range and 
4 degrees in azimuth. 


at short ranges is characteristic of ground clutter, and the 
random dots throughout the field of the picture are due 
to receiver noise. What we are attempting to do is to 
determine, from the number of pulses in any one range 
interval and in any one beamwidth, whether a signal is 
present, or if noise alone is present. 
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ANALYSIS 


Fundamental Relationships 


The probability density for the envelope R of a sine 
wave plus noise (P sinwt + J,), where the noise is con- 
fined to a relatively narrow band centered on the sine- 
wave frequency, is given by Rice as: 


RAR . ee + Ey (2) a) 
LG ERNE 

where Y is the noise power and J, is a modified Bessel 
function of the first kind and zero order. 

For a linear second detector, the density functions for 
the peak amplitudes of the detected pulsed signals in 
noise are also given by (1). For the more general case of 
half-wave detection, R can be replaced by the appropriate 
y = {(R)—for example, y = R’ for a square-law detector— 
to obtain the desired density functions.* 

When the quantizing in time corresponds roughly to a 
pulse width, the probability of obtaining a quantized 
video pulse, given a pulse of amplitude P in the IF 
amplifier, is 


p(k) = 


Pa= Pov) = He v dv exp ha, (2) 


where the variables have been normalized with respect to 
the noise power, L.e.,v = R/ V Wo , the predetection signal- 
to-noise ratio a = P/V , and V is the normalized 
amplitude threshold in the quantizer. 
_ In the special case where a signal is not present (a = 0), 
the probability of obtaining a quantized video pulse due 
- to noise alone is 


R= PO vy) = a v dv exp zh = exp =. (3) 


Having obtained the occurrence probabilities for 
quantized video pulses, the next step is to obtain the 
_ probability density for the sum of a given number of such 
_ pulses, that is, we ask the question, ‘‘What is the probability 
of obtaining exactly k successes (quantized pulses) in a 
set of m trials (number of repetition intervals over which 
we are integrating) when the probability of success in 
any one trial is p(P, or P,, in our case)?” This is essentially 
the definition of the well-known binomial distribution* 
| which is given by 


ae m! k/4 __ pp\m—k 


This then gives the probability of a counter reading of k 
out of a maximum counter reading of m. The probability 


28. O. Rice, “Mathematical analysis of random noise,” Bell Sys. 
Tech. Jour., vol. 23, pp. 282-332; July, 1944; vol. 24, pp. 46-156; 
January, 1945. 

3D. Middleton, ‘Some general results on the pee of noise 
through nonlinear devices,” Quart. Appl. Math., vol. 5, p. 446; 
1948. 

4W. Feller, ‘Probability Theory and its Applications,” John 
Wiley and Sons, Inc., New York, N. Y., vol. I; 1950. 
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of obtaining a counter reading of k or more is simply 


It can be demonstrated that the summation (5) is 
expressable in terms of the normalized, incomplete beta 
function which is tabulated.” Pearson’s definition of this 
function is 


/ a” (1 — 2)" dex 
0 


I,(p, ) a 
cel cook eee dx 


0 


In our case,” 
Q(k, mM, Pp) = Tim a 


These discrete distributions, while describing rigorously 
the statistics of binary integrations, unfortunately do not 
lend themselves too well to calculation. It is more con- 
venient to use the Edgeworth series approximation to the 
binomial distribution and then to obtain Q by integrating 
the approximate continuous distribution. Thus, the 
series that closely approximates b(m, k, p) is’ 


iy Cale a 


P,.{y) = Eo = 31e. Y) 4+ sel O16) ) 


ee (6) 


where 


the mean of the distribution = mp 


SSS) 
q Ss 
II ll 


the standard deviation = /mp(1 — p) 
| a, = coefficient of skewness = ee (6a) 
V mpl — p) 
: 6p — 6p + 1 
— 3)= coeficient of peakedness = 
( ) ‘ mp(1 — p) 
and 
i —y?/2 (n) dl” 
= =e y) = —7 oly). 
$0) = Ge 6 W) = FW) 


The desired cumulative probabilities may then be 
obtained by a termwise integration of (6), or 


Qk, m,) = f Pale) dy, (2 


where Y = k — m — 3/c. 


5K. Pearson, ‘Tables of the Incomplete Beta Function,” The 
University Press, Cambridge, England; 1934. 

OT St Mle Fano, “The Transmission of Information—II,”’ Tech. 
Rep. No. 149, Res. Lab. Elec., M. I. T., p. 21; February, 1950. 

190 (Oe Fry, ‘ rapepiy and Its Engineering Uses,” D. Van 
Nostrand Co., New York, N. Y., p. 256; ,1928. 
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10 2 5 y 
+ 6! asp” (Y) - = 
where 


: 
or) = | oway, 


and is a tabulated function along with ¢(y) and its first 
few derivatives.” The factor 3 in the lower limit arises 
by virtue of its allowing a better agreement between 
the cumulative probabilities determined from the discrete 
and from the continuous density functions.* 

A comparison of the results obtained from (5) and (7) 
is shown in Fig. 4 where the curves were obtained from 
tables’ of the incomplete beta function and the plotted 
points were obtained by using the indicated number of 
terms of (7). For large m, the density function (6) implies 
that the distribution tends toward a normal distribution; 
it is evident from the curves, especially for small p and k, 
that the higher-order terms in (7) must be considered. 
For large p fit with normal distribution is quite good. 


1.0 = —— =<— = 
A /\ Y\ Y\ p=0.9 
\| Q 
p=0.7 
\ DY L\ [\ 
0.1 ' p=0.5 
p=0.3 
Qi 
f\ A 
=0.1 
p=0.01 R 
0.0! | 
Y p=0.05 LEGEND 
O:USING 4 TERMS OF EDGEWORTH SERIES 
Q, X=USING 3 TERMS OF EDGEWORTH SERIES 
x O=USING 2 TERMS OF EDGEWORTH SERIES 
A=:USING | TERM OF EDGEWORTH SERIES 
(NORMAL DISTRIBUTION) 
0.001 
\e, 
x 
fa Q(k,m,P) 
m=16 
0.0001 
4 t 4 4 
0.00001 j 
(0) 2 4 6 8 10 12 14 16 
«— 


Fig. 4—Plot of Q; vs k for various numbers of terms and values of p. 


We have, then, all the mathematical apparatus on 
hand for calculating the detection probabilities, following 
integration, for signal plus noise, and for noise alone, as 
functions of the quantizer threshold V and the counter 
threshold K. To study the effects of these thresholds 


8 “Tables of the Error Function and Its First Twenty Deriva- 
tives,” Annals Harvard Computation Lab.; January, 1922. 
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on the detection efficiency, it is convenient to define a 
signal-to-noise ratio for the integrated signals and to 
maximize this with respect to V. 

The signal-to-noise ratio for signals and noise that have 
different probability distributions is, in general, not too 
significant a quantity from which detection probabilities 
can be deduced. This is so because a simple ratio cannot, 
by itself, specify the two distributions, and hence cannot 
specify the cumulative probabilities derived from them. 
This is unlike the situation in, say, an IF amplifier where 
signals and noise are linearly superimposed so that the 
distribution of noise on the signal is the same for any 
signal amplitude including 0. In this case, a signal-to-noise 
ratio does completely specify the two distributions for 
signals plus noise and noise alone. This predetector 
signal-to-noise ratio a is, consequently, a very significant 
quantity which is referred to throughout the analysis. 

Nevertheless, for purposes of expediency in analysis, 
it is desirable to define a signal-to-noise ratio for detected 
video. Where a reasonable amount of integration is used, 
such that the distributions approach the normal distri- 
bution, the signal-to-noise ratio can have appreciable 
utility. It cannot completely specify the desired detection 
probabilities, but from it a good approximation to these 
quantities can be obtained. 

While there are several ways in which one could define 
a signal-to-noise ratio for detected signals in noise, the 
particular definition that is most significant in our case 
is the following. The signal-to-noise ratio of the integrated 
video will be defined as the difference between the means 
of the distribution for signal plus noise and noise alone 
divided by the standard deviation of the integrated noise 
alone. Thus, from (6a) 


mP, — mPy 


(S/N) = 


where 
sae a Sarnia Se 
PES ee, 
From this it is evident that 
Jew eG: 
VPx(1 — Py) 


p 


is the equivalent signal-to-noise ratio p of the quantized 
video and that this is improved by the well-known square 
root of the number of samples taken. It is interesting to 
note that p can be determined directly from the discrete 
distribution of Fig. 1, that is, the corresponding mean for 
signal present is P, and that for noise Py , while the 
standard deviation for the noise is VPy(I — Py). 


The Effects of Quantization on Signal-to-Noise Ratio 


To determine the variation of p with Py , it is first 
necessary to obtain an expression for P, as a function of 


: 
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Py , that is, since P, and Py are both functions of V (the 
input threshold), it should be possible to eliminate V and 
Obtam P= (a.P). 

By making the substitution 


—y?/2 


p =e 
in (2), and remembering that 
IP => ert 
we obtain 
i _————— 
Pea tee? I.(a V —2 log p) dp. (9) 


PN 


Since (9) implies that the relationship between P, and 
Py is independent of V, it must also be independent of 
any function of V that might characterize a different 
detector. Thus, we reach the interesting and very useful 
conclusion that the P, vs Py relationship 1s independent of 
the detector characteristic, and is a consequence solely of the 
process of half-wave rectification and filtering without 
noticeable video narrowing. Our results will then apply 
for any half-wave detector of the form Z = f(v) for 
oe 0: 

Some useful series expansion of (9) may be obtained by 
substituting the series expansion for J, and integrating 
termwise; thus 


foo) 2k 
MY 


Ifa) = 1+ Le GBP? (t <o), 


and 


ON a2 log np) = 1+ 
using the relationship 
k : k-n k! n 
[ioe pdp = (1) 71 P 10g" P- 


After integrating and collecting terms, we find 


fee wie er cs (l= Py) 


ele 1 
ae te et eels Pye, 
2 
(2a aK 1 =F Py(— log lex) 
u 2 
= 91 Px(— log Py) har (10) 
or 
has at/inx isl: (ey 
Pe alee LOG ; 
where 


r, = E = 1 pa(—log Pa)* 
tao Kk} 


The coefficients \,, are functions only of Py . It may be 
shown that \,, must be less than unity, ie.,0 <A, < 1. 
For large n, X,, — 0, for all Py , so that the series con- 
verges fairly rapidly. 
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An alternative expression for P, is obtainable by 
regrouping the terms in powers of (— log Py); thus 


P,=Py+(Q0—- e* ”)Px(—log Py) 


1 —a?2 a —a? 
}- (1 nC, be a 9 é \ Plog Ne + ol aad ) (11) 
or 


P= Pr toiaa ot onl ae 


where 


nih 1 a k 

= 1 = One iis @ A 
Us hone 
the coefficients yu, are functions only of a. Also0 < u, < I 
and, for large n, u, — 0 for alla < ©, which again indicates 
fairly rapid convergence. 

Still another form of (10), particularly useful for the 
small-signal case, is 


2 2\2 
—_ a —a?/ e N 2 1 a —a?/2 
P= Py 9° *Py(—log Py) 4 3 (< ) € 


(Px(—log Py) oe + Py(—log Py) a hax (12) 


An expression for p can then be obtained by using any of 
these and (8). Thus 


1-="Py pee Ne € ) 
es zn 2 | — 13 
Py VPy( — Py) = 2 cy 
or 
2 eles ey aires (14) 
VP x1 = Py) n=1 n! 


These are very useful and rigorous series for the effective 
signal-to-noise ratio of binary quantized signals. The 
expansions are made in terms of the predetector signal-to- 
noise ratio and the quantizer threshold as specified by the 
probability of quantizing noise. Furthermore, they hold 
for any half-wave envelope detector. The variation of p 


P = EQUIVALENT S/N RATIO AFTER QUANTIZING 
a = PREDETECTOR S/N RATIO 
P,= PROBABIL!ITY OF QUANTIZING NOISE PEAK 


Pp a? 
— <<! 
ae FOR 2 


0 0.001 
Fig. 5—p vs P, for constant a. 


with Py for various a is shown in Fig. 5. It is evident from 
these curves that there is an optimum setting of the 
input threshold (i.e., optimum Py) for any given a to give 
a maximum p. For a = 1, which ordinarily corresponds to 


6 IRE TRANSACTIONS—INFORMATION THEORY 


the minimum detectable signal in any practical post- 
detector integration system, the optimum Py = 0.1. For 
larger a, the optimum P, decreases, but Py should be set 
for best results on the weakest signal of interest. It is 
further evident from the curves that, for large a, p does 
not increase indefinitely, but reaches a limiting value 
which is a function of Py , and is given by the first term 
in (13). It is also obtainable by realizing that, for large a, 
P, — 1, in which case (8) becomes 
tor Os ie (15) 
This saturation effect on p is not a serious limitation in 
our case since, for Py = 0.1, pmax = 3, Which is sufficiently 
high to insure good detection even without integration. 
For the small-signal case, i.e., a’/2 « 1, the first few 
terms of (12) give a good approximation to P, . Thus 


2 2 
P,@~Pyt+ (“\px(—log Py) for a <<a (16) 


Hence 


a, Pw (—log Py) for Oe (17) 
Soe Nie ay 5 


Differentiation of this, with respect to Py , indicates a 
maximum at Py = 0.203. Hence, for very small signals, 


2 
* (0.804) at Py = 0.203. 


Pmax = 9 (18) 


We conclude then that the optimum setting for Py 
approaches 0.203 as a diminishes. A plot of (17) is shown 
in Fig. 5 and the agreement between it and the accurate 
contour for a = 1 is quite good. 


Determination of Best Counter Threshold Kk 


Having shown that, for best weak-signal detection, Py 
should be set for 0.1—i.e., a = 1 is ordinarily the weakest 
signal detectable with a reasonable number of hits per 
beamwidth—the problem is to select K so that the allow- 
able false-alarm probability Qy is met. An approximate 
figure for Qy = (Qy) can be calculated by assuming that 
the integrated noise will be approximately normally dis- 
tributed. Thus from (7) 


Qx = 4) bee ¢ (Yy), 


3 K — My — 3 
Y= eee 
on 


If we define the minimum detectable signal as one 
detected 50 per cent of the time, then, for the signal, 
Y, = 0, or K — (1/2) = mi, . Under these conditions, Y x 
signifies an integrated signal-to-noise ratio p,y, since 


= Vp 
For a desired false-alarm probability Q, , the tables 


yield a corresponding value for Yy , ie., if Yy = 4, 
Qye=8. x 10% Then 


(19) 
where 


ee Mie = Wr; (2 0) 


On 
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( G 2) — (Pout) V mP x(1 The Py) + mP x 5 (21) 

For Py = 0.1, this reduces to 
K = (pou)0.3V m + 0.1m + 3, (21) 


from which Table I, for best counter threshold settings, 
is obtained. 


TABLE I 

(etaxtni, 4 Pout == 5) 
m Qn =o 10m Qn <al|Om® 
4 3.8 3.9 
8 4.7 5.6 
16 6.9 8.1 
32 10.5 12.2 
64 16.5 18.9 


For convenience when using a binary counter, these 
would ordinarily be rounded off to the nearest power of 2, 
incurring a very slight loss in sensitivity. 

The precise false-alarm probability Qy can be determined 
from (7), which for Py = 0.1 becomes 


oud = 3 (=F — Wood + i (B) 


L® 10 e i 
p (Pour) oF 6! 3 m 


With p,.. specified, then for a given number of integrations 
m the value of p is also specified, which by the relationships 
given previously yields a value for a,;, , the minimum 
detectable signal. The curves of Fig. 6 indicate the 
variation of Gni, With m (the number of samples inte- 
grated). It will be noted that the minimum detectable 
signal decreases very slowly with an increase in the 
number of integrations since dp;, «(1/4V m). This 
fourth-root relationship for small signals follows from 
(17) and (8) since pu, = V mp, and p«a’ then for constant 
p,a «(1/4Vm). 


Qv =1- 


psn) aro (22) 


= NUMBER OF ADDITIONS 
= PREDETECTOR S/N RATIO 
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,o 3 = PULSE WIDTHS PER QUANTIZED 
INTERVAL 
a 
8=6 a MIN 
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31x 
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Fig. 6—k/m and minimum a, vs m. 


Comparison with Analog Integration 


A comparison of binary integration with analog inte- 
gration can be reduced to a comparison of the effective 
signal-to-noise ratios prior to integration in the two cases, 


1955 


since for both cases the effect of integration is to multiply 
this input signal-to-noise ratio by +1/m. The two must 
be defined, however, in the same manner, and we shall 
define the signal-to-noise ratio for the continuous dis- 
tributions dealt with in the analog case in the same manner 
as we did for the quantized case—namely, as the difference 
between the means of the signal and noise distributions 
divided by the standard deviation of the noise alone. It 
may be shown that, for the continuous distributions, the 
postdetector signal-to-noise ratio m/o(0) is given’? by 


m 1 k a\” 
— Be: ) Le =) ear |, TB} 
To) BE oat ane | 2 2 oD 
(+8 


where k is the power of the detector law and ,F, is the 
confluent hyper-geometric function.” 
For the small-signal case, a°/2 « 1, this becomes 


k 
mae B a (24) 
F (0) Td + k) er 2 
r(1 + t) 


which, for the linear detector (k = 1), reduces to 


= 2 
ssblaaey 0.960($-). 


J (0) 2 


(25) 


For the square-law detector (k = 2), (23) reduces 
without approximation to 


2 
m a 


oma) yy 


(26) 


In general, the coefficient of (a’/2) in (24) has a maximum 
value of unity for k = 2 and falls off slowly for all other 
values of k. 

In the small-signal case for quantized signals, we recall 
from (18) that 


p= 0.804(%). (18) 

A comparison of (18) with (25) and (26) leads to the 
conclusion that the effect of quantization on the signal-to- 
noise ratio is to reduce it by a factor 0.838 or 1.54 db in 
comparison with the signal-to-noise ratio from a linear 
detector, and 0.804 or 1.92 db in comparison with that 
from a square-law detector. Since the square-law detector 
yields the highest signal-to-noise ratio for small signals, 
the maximum loss sustained is of the order of 1.9 db. 
It should be remembered, however, that most analog 
integrators are nonideal, i.e., the signals decay in the 
memory, and losses of 1 to 2 db in output signal-to-noise 
ratio from this cause are not at all uncommon in practical 
devices of this type. It may be stated, therefore, that, 

°J). Middleton and V. Johnson, ‘“‘A Tabulation of Selected Con- 


fluent Hypergeometric Functions,’’ Tech. Rep. No. 140, Cruft Labs., 
Harvard University; January, 1952. 
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while analog integration starts off with an advantage of 
1.5 to 2 db in input signal-to-noise ratio, the perfect 
memory of the binary integrator makes up for this loss 
and the two systems should be about equivalent in final 
detection efficiency. 

A comparison of p and (m/°(0)) for the linear detector is 
shown in Fig. 7. In the weak-signal region, a < 2, the 
difference is small. Both, of course, suffer markedly from 
the small-signal suppression effect—i.e., signal-to-noise 
aa’ for small signals—as a comparison with the pre- 
detector signal-to-noise ratio a will demonstrate. 


6 


S/N RATIO 
ow 


° \ 2 3 4 5 
a 


Fig. 7—Comparison of predetector, postdetector and postquantizer 
signal-to-noise ratios. 


The Effect of the Width of Quantizing Interval 


The previous analysis was based on having either a 
signal pulse or a noise pulse in the quantizing interval. 
Now we turn to the more usual case wherein the quantizing 
interval is several, say, 5 pulse widths in duration. If 
independence is assumed between the signal levels one 
pulsewidth apart (an assumption that the usual match 
between circuit bandwidth and pulsewidth makes reason- 
able), then the occurrence probabilities for the quantized 
signals and noise P{ and Py are related to their former 
values by 


qd 7 Je) = (1 =i JOG. rs, Py (27) 


(ie Pir =e): 


where, in the signal case, it is assumed that there is one 
signal pulse of probability P, and (6 — 1) noise pulses of 
probability Py within the quantizing interval. As in the 
earlier case, 
aes EN, 
p — = eee (28) 
i V/ PAO <2 Py) 


A comparison of ps; with p, can be made on many bases; 
and the easiest, although not the most significant, com- 
parison to make is on the basis of constant Py , i.e., same 
input threshold setting, independent of 6. From (27) 
and (28) 
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P, — Py ipa epee noise ratio a, the quantized signal-to-noise ratio is that 
ee Vi RGR i ae P 3 much poorer. Because of the detector loss, however, 
” * % wherein p <a’, the ratio of the minimum detectable signals 
IP. (1 3) ye for 6 = 1, and for 6 = 6, all other things being equal, is 
JEN mE N Sy 
= pv ae (29) +/3.44 or 1.85. 
Vices: 
For small Py such that 6 Py « 1, this becomes = EQUIVALENT S/N RATIO AFTER QUANTIZING 
= PREDETECTOR S/N RATIO 
1 = PROBABILITY OF QUANTIZING NOISE PEAK 
[hn ner (30) = NO. OF PULSE WIDTHS PER QUANTIZING INTERVAL 


This result agrees with the intuitive feeling that, since 
there is 6 times as much noise power now in the quantized 
interval, the corresponding signal-to-noise ratio should 
be 6 times smaller. For larger 6 or Py , the ratio of 
ps/p, becomes much smaller than vi and approaches 
Oas Py > 1. 

The more realistic basis for comparing p; and p, is that 
of a constant Px , that is, on the basis of a changing thresh- 
old (Py) as 6 changes to keep constant the average 
number of noise pulses (Px) being counted. This is ordi- 
narily the way any system for a given number of additions 
and a specified false-alarm probability would have to be 
adjusted. Eliminating Py between (27) and (28), we find 


res 
(US = x Pe [1 


where P, is given by (10), (11), or (12) with 
Py =e Bae): 


(= Pir Ge.) eno) 


For large signals (large a), P, — 1, and 
ew 
(dy SS ee ; (32) 
N 


which is precisely the same result we obtained for p, . 
In the small-signal case, (31) reduces to 


ey eae 
Pia 2 Px 


{(l-— (1 — PS? ][-log 1 — (1 — Ps))]'”}; 


C1. (33) 


A further simplification of (33) can be made for small 
Px . Thus, if Px/é « 1, then 


Px —log Py 


2 
pens iberse EL) 


The variation of p; with Px as calculated from (33) for 
various values of 6 is plotted in Fig. 8. The general nature 
of all these curves is much the same; the magnitude of 
p; , of course, reduces for increasing 6, and the maxima 
shift toward higher Py . For 6 = 6 and Px = 0.1, the 
ratio of (p;/p;) is 3.44. However this does not mean that 
the minimum detectable a is 3.44 times smaller when 
6 = 6; it means that, for the same predetector signal-to- 


fo) 0.001 0.01 0.1 1.0 10 
Ph 


Fig. 8—p vs P, for various a, a?/2 < 1. 


Effective Signal-to-Noise Ratio for Signals of Varying 
Amplitudes 


When the signals to be integrated are derived from a 
radar with a scanning antenna, their amplitude is not 
constant as assumed in the previous analysis, but varies 
with time [a = a(¢)]. On the average, the envelope of the 
set of signals from a given target will resemble the beam 
pattern of the antenna, and the occurrence probability for 
the corresponding quantized video pulses will vary ac- 
cordingly. The mean of the integrated signal distribution, 
instead of being mPs as before, will now be given by 


™m 


in, = Dy Be = mP, 


where 


m 


Dae 


P, = + 
Ue FER 


From (10) and (13), therefore 


Be 2 et ! MeL (Vee 
és i VE Dyes 
(36) 


where a; = a(t;) is the predetector signal-to-noise ratio of 
the jth sample in the set of m being counted. Eq. (36), 
then, is the desired expression for the quantized signal-to- 
noise ratio in the case of nonuniform signal amplitudes. 
It is, however, a rather cumbersome equation to use, and 
fortunately in the small-signal case (36) simplifies to: 


Cae Ga ie eee a” 


5 (37) 


where 


and is the mean square signal-to-noise ratio of the set of 


m samples. The results given previously on counter 
thresholds and optimum Py will still hold for small signals. 
To get the minimum detectable a, however, a” instead of 
a must be used in the varying-signal case. 


RELATIONSHIPS TO More-GENERAL Daerection THEORY 


It should be mentioned here that this report describes 
an analysis of a particular method of signal detection 
where, for the sake of circuit economy, particular criteria 
are used to determine the presence or absence of a signal. 
The statistical criterion used is essentially the Neyman- 
Pearson one, and the question of whether or not this is 
the best possible criterion has not been considered. 

The general theory of testing samples to determine 
whether they fall under hypothesis H, (in our case, noise) 
or under hypotheses H, (in our case, signal) is one that 
has received a great deal of attention in the field of quality 
control and lately in the field of communications. In 
testing for hypothesis H, , two types of errors are con- 
sidered. These are: a Type | error, having probability a, 
in which H, is judged to be true when it is not (A, is true); 
and a Type 2 error, having probability B, in which H, is 
judged to be false when it is not (H, is false). 

It may be rather loosely stated that the general object 
of the testing procedure is to minimize a and B—that is, 
to maximize the probability of being right. Various 
methods for examining an ensemble of samples to judge 
their nature have been proposed, and recently an extensive 
analysis has been made by Middleton of three such 
methods which he applied to the detection of signals in 
noise.’° The criteria differ primarily in the manner in 
which a, B, and n (the number of samples) are varied or 
held constant for a given predetector signal-to-noise 
ratio a. 

For instance, the Neyman-Pearson observer holds a 
and n constant and produces a variable B with varying a, 
while the sequential observer fixes a and B, and uses a 
varying number of samples n for varying a. In the case 
analyzed in this report, we considered integration of a 
fixed number of samples and adjusted the thresholds to 
obtain an acceptable false-alarm probability Qy. The 
detection probability Q, then varied with a and this, of 
course, is characteristic of the Neyman-Pearson observer. 
The relationships between a, B, and our Qy, Q, are 
simple, and are given by 


Qy , (38) 


Ti Oise 


In the discussion of these three observers, Middleton” 
employs the notion of a betting curve where the betting 
function is defined as 


aa) = "pl — a) ++ 'q(1 — B); 


p and q = (1 — p) are essentially arbitrary weighting 
factors which assess the importance that H, or H, , 


a= 


B= 


(39) 


0D, Middleton, “Statistical Criteria for the Detection of Pulsed 
Carriers in Noise,” AF Cambridge Res. Center Rep. No. E-5091; 
August, 1952. 
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respectively, are’ true. If, over a large number of tests, 
signals:and noise are equally likely, and the determination 
of both is equally important, then p = q = 1/2. The 
betting function may be interpreted as the probability of 
making the correct decision no matter which of the two 
hypotheses actually obtains. 

In our analysis, we have derived expressions for Q, and 
Qy from which a curve of Q, as a function of Qy and a could 
be obtained. This would have the general character of 
that in Fig. 9. The entire curve has not been calculated, 
however, but we have confined our attention to two 
points on it. In a practical sense, since the curve is neces- 
sarily a monotonically increasing function, these two 
points pretty well specify the function. These points are 
Qy at a = 0, and Q, = 0.5 at a = a,;, CIII-A). The 
relationship between detection probability curve and 
betting curve is readily obtained from (38) and (39); thus 


w(a’) — pQw i 
q 


Since p and Q,y are constants in our case, Q, is linearly 
related to the betting function and the curve of Fig. 9 
can be interpreted as a type of betting curve. 


1.0 


0.5 
Qs 


Fig. 9—Detection probability as a function of predetector signal-to- 
noise ratio. 


While the question of which of the statistical criteria 
gives the “best”? results has not been treated in this 
report, there is some evidence’ to indicate that the 
Neyman-Pearson observer in the radar case performs 
about as well as and, in some cases, slightly better than 
the other observers. Hence, in addition to being the 
easiest criterion to implement, it is also likely to be one 
of the most sensitive. A possible reason for this is that 
the decision made is based on the total (integrated) 
received energy. In a physical sense, one intuitively feels 
that if the total signal energy is taken into account, the 
detection efficiency will not be significantly different in 
any of the statistical detection schemes. 
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An Expansion for Some Second-Order Probability 


Distributions and its Application to Noise Problems’ 


J. F. BARRETT} AND D. G. LAMPARD}] 


Summary—In this paper it is shown that, in general, second-order 
probability distributions may be expanded in a certain double series 
involving orthogonal polynomials associated with the corresponding 
first-order probability distributions. Attention is restricted to those 
second-order probability distributions which lead to a ‘‘diagonal’’ 
form for this expansion. 

When such distributions are joint probability distributions for 
samples taken from a pair of time series, some interesting results 
can be demonstrated. For example, it is shown that if one of the time 
series undergoes an amplitude distortion in a time-varying ‘‘instan- 
taneous”’ nonlinear device, the covariance function after distortion 
is simply proportional to that before distortion. 

Some simple results concerning conditional expectations are 
given and an extension of a theorem, due to Doob, on stationary 
Markov processes is presented. 

The relation between the ‘‘diagonal’’ expansion used in this 
paper and the Mercer expansion of the kernel of a certain linear 
homogeneous integral equation, is pointed out and in conclusion 
explicit expansions are given for three specific examples. 


INTRODUCTION 


NHE SOLUTION of most problems involving noise 
in nonlinear devices is difficult. However, a few 
years ago Bussgang’ demonstrated an interesting 

and useful result. He showed that, if one of a pair of 
stationary time series with Gaussian probability dis- 
tributions was amplitude distorted in a fixed “instan- 
taneous” nonlinear device, then the cross-correlation 
function after the distortion is proportional to the cross- 
correlation function before the distortion. 

An attempt to extend the scope of Bussgang’s results to 
distributions other than Gaussian, led to a class of dis- 
tributions which is discussed in this paper. Such dis- 
tributions exhibit a number of interesting properties some 
of which are studied here. It is believed that many results 
will be useful in dealing with certain noise problems. 


ANALYSIS 


We suppose that p(a,; ; x.) is a second-order probability 
distribution. The corresponding first-order probability 
distributions are given by 


pila) = / P(X, ; Xa) dx, 
A pee é (1) 
De = i DUA ep) dex | 


*Some of the material of this paper is taken from a Dissertation 
submitted by D. G. Lampard for the Ph.D. degree at the University 
of Cambridge. 

+t The Engineering Lab., Cambridge, Eng. 

t Dept. of Electrical Engineering, Columbia Univ., New York, 
N. Y. Formerly at the Engineering Lab., Cambridge. At present on 
leave from CSIRO Diy. Electrotechnology, Sydney, Australia. 

1J. J. Bussgang, ‘‘Cross correlation Functions of Amplitude Dis- 
torted Gaussian Signals,’ Tech. Rep. No. 216, Res. Lab. Elec., 
MIT; March, 1952. 


We construct’ two sets of normalized orthogonal 
polynomials @,"’(#,) and 6,,”’(a,) such that 


| 
or 
= 
3 


th DilepOgs (tO, iy) da. = 


/ P2(X>) O8 (Bs) On (es) dx» = Onn 

We now expand the second-order probability distri- 
bution in a double ‘Fourier’ series involving these 
polynomials. Thus 


p(x, ein) == Di (Gi) Po(Ge) E, Sy CS Ge) kD (3) 
m=0 n=0 

The coefficients a,,,, may be determined in the usual way 

by multiplying both sides by 6,’ (x;) 0;°’ (a2) and integrat- 


ing using the orthogonal property (2). We find 


Ona I p(t, 3 >) Oy (41) 0, (to) Ax, Axe . (4) 


Thus we see from (3) that the second-order distribution 
is now determined completely by the two first-order 
distributions and the coefficient matrix [a,,,,]. 

In this paper we restrict our attention® to the class, A 
say, of distributions p(a, ; x) which have the property 
that the matrix {a,,,,] is diagonal. As we shall see from 
examples, some important distributions that occur in 
physical problems are in this class. 

Thus for all p(a, ; x2) in A, (3) may be written* 


p(x, ; 433) = Dia) po (Lo) SS GO (aay Gg Ge) (5)A 
n=0 
where the coefficient a, is now given by 
a, = ib (tr 3 2) 4, (41) 0,” (a2) dx, dr» . (6) A 


w.r. 


We now determine the form of the polynomials of 
degree 0 and 1. As p,(x,) and p(x) are probability dis- 
tributions we must have 


ib pile) -1-V-de, = 1 (7) 


> By applying the Gram-Schmidt procedure to the sequence 
1, x, x? --- . For details see Courant and Hilbert, “Methods of Mathe- 
matical Physics,” Interscience Publishers, Inc., New York, N. Y 
vol. I, p. 50; 1953. . 

*The authors have not been able so far to find what general 
ie ere must be placed on p(a ; 22) in order that it may belong 
0 A. 

‘Throughout this paper, equations which are only true when 
p(x: ; £2) belongs to A, will be denoted by having A added after the 
equation number. 


— 1955 


igs pi(a1) (a; — wi)-1-da, = 


| 
a) 


(8) 


| 
= 


ae Pi(4s)(%1 — pa)(%, — pr.) day = of (9) 


with corresponding results for p,(x.). Here we have used 
the symbols uv, and o, for the mean and standard deviation 
respectively. It follows that 


eG) als 65” Go) 


1 


; (10) 
((a,) gta ales 


; Lo — 
{ A?) =, © Mes 


O71 02 


On using (10) and (6), it is easy to see that the co- 
efficients a) and a, are just 


Gna ul (11)A 


Mi Se ta) ig M2)) eat 
7102 ae 


= (12)A 
That is a, is just the normalized covariance’ or correlation 
coefficient. 

We now prove an important inequality® for the co- 
efficients a, . Let \ be a real variable and consider the 
expectation 


((6, (a) 5 NACA 


= ff ple 5 20) + OIF de dey (13) 
= ff peer 5 ea 0i? ey des de 

+ 2d if (i Plas ; %2)O, (21) 0, (Xs) dx, dat, 

$n? ff vl se) P dry dey. (14) 


w.r. 


On introducing the expansion (5) for p(a, ; 2) and carrying 
out the integration, making use of the orthogonal proper- 
ties (2) we obtain 


(6 (a3) FAO Ga) P) = P+ 2ra, +’. 


As the expression on the left-hand side must be positive 
for all real \ we must have 


nls 


(15) A 


for all n- (16)A 


which is the required result.” 


APPLICATION TO TIME SERIES 


In the remainder of this paper we shall think of #,; and 
Z_ aS being sample values from a pair of time series 


>Cramér, ‘‘Mathematical Methods of Statistics,” 
University Press, Princeton, N. J., p. 265; 1946. 

6 This is just the Schwarz inequality adapted to this particular 
problem. See for example, Lovitt, “Linear Integral Equations,” 
Dover Publications, New York, N. Y., p. 125; 1950. 

7The authors are grateful to S. O. Rice for pointing out these 
inequalities in a private communication, 
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X(t), X(t) at times ¢, and ¢, respectively, as it is this 
interpretation which has most physical significance. 
Thus we write 

y= X(t) : (17) 
Le = @,(1,) 


We then consider an ergodic ensemble of such pairs of 
time series and take p(a, ; x.) to be the second-order 
probability distribution of this ensemble. In general this 
probability distribution will be a function of t, and ¢ . It 
then follows that the coefficients defined by (6) will be 
functions of ¢, and t, so we may write 


0, = Obert) (18) A 


In particular if the time series is stationary, only time 
differences are significant so that we have 


A, = a(t — t). (19) A 


The first-order probability distributions p,(z,) and 
p2(%2), [and the corresponding sets of polynomials 0,” (2;) 
and 6,” (a»)], will also, in general, be functions of ¢, and & , 
but in the stationary case will be independent of time. 


XC, (4) y, 


Xx. f { x} ¥@ 


Fig. 1 


Noise In “INSTANTANEOUS” NONLINEAR DEVICES 


Let us consider the system shown in Fig. 1. 
The input-output relations of this system are given by 


yilt) = x,(t) 
y(t) = fix} 


where the function f is assumed not to involve differential, 
integral, or delay operators. In general f may be an 
explicit function of time, assuming that such temporal 
dependence is statistically independent of the input. 

A device characterized by such a function f will be 
called®* an ‘“‘instantaneous” time-varying nonlinear device. 

Let us now find the (un-normalized) covariance function 
for the outputs. We have 


(20) 


Wioltr ; to) 
= (lyilh) — Gilt) Ilye(e) — (Yo(te))]) 
= ([a. — mwil[f(v2) — (f(x2)))) 


(21) 
(22) 


§ The term ‘‘zero-memory”’ is also commonly used in place of 
“Instantaneous.” 
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= [| (t, — ws) { f(t) — (f(ee))} ple ; 22). dx, dx. (28) 


Wiekl 


| 


Oo} l| 6; (a1) { f(ae) — (f(x2))}p(ay 322) dx, dx. . (24) 


To make further progress, we now expand f(x.) in a series 
of the polynomials 6,,”’ (v2). (We assume that the expansion 
can be justified.) Thus 


ey) =, Ss Gi One (es) (25) 
m=0 
where the coefficient c,, = ¢,,(t2) is given by 
Cn fa / f (2) pole) 6.” (20) dx» ° (26) 


In particular, remembering from (10) that 6,” (v2) = 1, 
we see that 


Be [ re. ihn ee len). (27) 
Thus ‘sh 
fle) = Sed) = Dende). (28) 


Thus using (28) in (24) we have 


Wi 2(b ; ty) 


= g, I wend cn 05a) pole SOLE hia (048) 
m=1 
We now use expansion (5) for all p(x, ; 22) in A and find 


Wiolti 3 be) = 14 i 0; (&1)p(a1)pa(r2) 


: ys Ss CmOn 6, (24) Oe (25) ae (2s) dx, dis (30) A 
m=1 n=0 
= 010 ,C, ) (3 1) A 


where, in carrying out the integrations, use has been 
made of the orthogonal properties (2). 

We note from (12) that a, is just the normalized 
covariance of the inputs, that is 


Ae (a, Sao e Ls)) a Vroltr 3b) 


(82) A 
0102 092 
Thus (31) may be written 
ae by } t) = Cra AG ; faye (33) A 
when 
ei — ; (ass ea) q 
C = C(t.) = (a2) rae Ax» . (34) A 


For stationary time series and fixed ‘instantaneous’ 
nonlinear devices (33) may be written as 
W2(7) oa C'- Pro(7), 


where the constant C is independent of time and when we 
have written 7 = ¢, — ¢t, . The y and W are just the ordi- 
nary cross-correlation functions of the fluctuating parts 
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of the input and output respectively, of the system of 
Fig. 1. In the still more special case in which the input 
time series are not only stationary but identical, that is 
v(t) = x,(é), the result (35) becomes identical to that 
obtained by Bussgang” for Gaussian noise. 

Luce’ has demonstrated that Bussgang’s result applies 
to a certain class of distributions, of which the Gaussian 
distribution is a special case. However Luce’s distributions 
are of a fairly simple form’’ and it is easy to show, by 
means of specific examples, that there are distributions 
in A which are not of the type considered by Luce. 


Somr ReEsuLTS ON CONDITIONAL HXPECTATIONS 
For a distribution in A it is easy to see from (5) that the 


conditional probability density for x, given 2, is 


20112) — play Tomeepaed. COA 


pi(21) 


Then the conditional expectation for 6,’ (a2) given a, is 


CSAC) | a1) 


2) 


= An O (23) G 


D(x» | a) = 


On; (£2 ¥Do(xs) Ss a9, (x1) 0 es) dx» (37)A 


(38) A 


If in particular we take m = 1, we have, using (10) and (12) 


SY moe vy 
Gan ») = p(t ; ts) ei 
Oo o 


1 
When 2,(¢) = 2,(t) is a stationary time series with 
normalized auto-correlation function p(r), (39) becomes 


Pa) (40) A 
t, — t, . This result 


(39) A 


aes ae 1D | a) = p(t) (x, ri 


where, as usual, we have written 7 = 
has an obvious interpretation. 


STATIONARY MArRKOvV PROCESSES 


It has been shown by Doob” that a stationary one- 
dimensional Markov process with a Gaussian probability 
distribution must have a correlation function which is an 
exponential function of time. In this section we show that 
this result is true for any distribution in A. 

Let x, , %2 --- 2, be samples from a stationary time 
series at times ¢,, ¢,, --- ¢,.° If this time series is a 
Markov process then 


Des | vy ) LES SO Cit) a D(Xp | Tw (41) 
by definition of a Markov process. Thus 
P(x, ? Li © One Ln) = Des 5) £5) ) 
POU te teans;) P(X,-1) ) 


® Bussgang, loc. cit. 

oR, Luce, ‘“‘Amplitude Distorted Signals,’ Res. Lab. Elec. 
Quarterly Progress Report, p. 37; April 15, 1953. 

1A simple change of variables always reducing the second-order 
probability distribution to the product of independent first-order 
probably. Cleve ution 

2 J. L. Doob, “Brownian motion and stochastic i uo 
Math., vol. 43, p. 351; 1942. es ge Se 


13 It is assumed that th > tr1 > +++ > hh. 
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Gow A ae age P(ln-1-5 Xn) (43) re P(t. , Zo) = pte , 1). (53) 
P(%»-1) Then the expansion (5) may be written 
i PGi wes) Pp Gor 2s) = Pei, L) (44) oo 
Plo) +++ P(Xn-1) D(X , C2) = p(ai)p(e2) DY a,8n(@1) O(a). (54) A 
by induction. i: 


In particular we find 
PLr_, L2)p(t2 5 Xs) _ 
P(x2) 


If the two-dimensional distributions p(x, ; x2) is in A, 
we can use the expansion (5) in (45). We find™ 


p(x, , Lp , Xs) 


Digi ke , x3) = (45) 


DS Om(t, — tian(ts — ty) 


m=0 n=0 


“O,,(%1) Op: V3) On|) On(ces).. (46) A 


Let us now find the two-dimensional probability density, 
p(x, , x3) by integrating out x, from both sides of (46). 


= p(x1)p(x2)p(xs) 


Wi aes, La) = he [LGR Matinee eon ike (47) 
= p(x1)p(«a) Bs OAts — b)a,(t; — 4))6,(a)) 0. G2). (48) A 
But as p(a, , 73) is an A we may write 
p(t , %) = pla,)p(as) d ail — 4) (21) O(a). (49) A 
For (48) and (49) to be consistent we must have 
Gn(tz — ty) = an(ts — t2)a,n(t. — th): (50) A 


It is clear that the only continuous solution of this 
equation is of the form 


ar) = aes 


where £8, is a constant depending on n. 
In particular when we remember that a,(7) = p(7), the 
normalized auto correlation function, we see that 


(S1)A 


Mayen, 7 > 0, (52) A 


thus establishing the required result. 


AN INTEGRAL EQUATION RELATED TO THE 
EXPANSION (5) 


The obvious similarity in form between the expansion 
(8) and the Mercer expansion” for the kernel of a linear 
homogeneous integral equation suggests that there may 
be an approach to our problem which is based on the 
theory of integral equations. 

To demonstrate this connection we shall assume 
p(x, , 2) to be a symmetric’® two-dimensional probability 
distribution. That is, we assume 


14 We have written dn(te 
avoid confusion. 

% Courant and Hilbert, loc. cit., p. 138. 

16 This symmetry restriction can be removed by using theory of 
1djoint orthogonal functions. Courant and Hilbert, loc. cit., p. 159. 


— t), ete., in place of our more usual a, to 


Multiplying both sides by 6,,(a.) and integrating with 
respect to x, we have, using the orthogonal property (2) 


iP P(r 5 L2) Om (a2) day = Amp(1)Om(ar) (55) A 
which ae ee 
ie vase Ste | {-V ples) On(02)} da 
= dn{ V p(t) On(as)} (56) A 


which is now in the standard form for a linear homogeneous 
integral equation 


it K (a1 , L2)bm(X2) dz = Amm(a1), (57) 
with 
KG ee 58) A 
Tne ee Be 
Pm(x) = V p(x) On(x) (59) A 
Ne = Oe - ; (60) A 
It follows from the definition (2) for 6,(a) that 
J baladou(a) de = f pla) 8m(2) 6,(2) dt = Oy (GI)A 


so that the ¢,, defined by (59) are normalized orthogonal 
eigenfunctions of the linear homogeneous integral (57) 
with corresponding eigenvalues \,, given by (60). 

Then, providing it can be shown that the set of eigen- 
functions defined by (59) is complete, the equivalence of 
the expansion (54) and the Mercer expansion 

K(x, , X2) = Zs An Pn(Li)Pn(X2) (62) 
is demonstrated. 

It must be pointed out, however, that the integral 
equation approach does not seem to be a very suitable 
starting point because of the difficulties of justifying’ 
the Mercer expansion and of showing when the eigen- 
functions are of the form (59), which is essential for the 
applications discussed in this paper. 


SoME EXAMPLES OF DISTRIBUTIONS IN A 


In this final section we give three examples of second- 
order probability distributions which are in the class A. 
We shall choose distributions which are “symmetric”? in 
the sense of (53) in order to keep the details of the analysis 
as simple as possible. 

1 Sufficient, but not necessary conditions are the continuity and 


definiteness of the kernel K(a, 22). Courant and Hilbert, loc. cit., 
p. 138. 
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Example 1 


The second order Gaussian probability distribution has 


the well known form*® 
fxs te =o eel 2, —1/2 7 i xy + v3 7x ape 
P(X, , X) = ae | p) exp 1 ol — p) 


where we have written co, p for the standard deviation and 
normalized correlation function respectively. 
The corresponding first-order probability distribution 


is 
1 deze 
p(x) = WA oe ‘-4 a. (64) 
We now make use of Mehler’s expansion,” namely 
2\—1/2 1 Daa a 73) ~ 2pants| 
lta ” exp — { 2 
(Cigseen) Pics ie 
— n TIA ) EG) ind 
SEU Dia = eo eames Ce 


where H,(x) is the Hermite Polynomial of degree n 
defined*’ by 
d" ( ,—x2/2 


dx" ¢ 


Then using (65) it follows that (63) may be written 
coo phe oe ea es 
AE NEE = Gy BN dat) o 


Sp Hall Hele) gy 


7 
=A n! 


= (-1)"H,(xe"'”. (66) 


The Hermite have the orthogonal 


property” 


Polynomials 


(68) 


= 6” “HH GAs) dr =o)! 
Vin bs a 
so it is seen that (67) can finally be written in the standard 
form” 


ple 5 22) = pler)plas) D2 a,0,(e:) Oa) (69) A 
acre 6,(c) = @!) “*A,@) (70) 
a, =p 


showing that p(a, , ®2) is in A. 


Example 2 


When Gaussian noise is applied to a narrow bandpass 
filter, the output of the filter behaves as a sine wave at 
the midband frequency with random, relatively slowly 
varying amplitude and phase. Rice has shown” that the 


6S. O. Rice, Bell Sys. Tech. Jour., vol. 23, p. 50; 1945. We have 
assumed zero means in this example for simplicity, 

1G. N. Watson, “Generating functions for polynomials II,” 
Jour. London Math. ‘Soc., vol. 8, p. 194; 1933. 

0 Cramer, loc. tt., Ds 133. 

21 Cramér, loc. cit., p. 133. 

2 This expansion isa fairly well-known one (e.g. Cramér, p. 133) 
and has been used by Sie »gert in noise problems; see “On the evalua- 
tion of noise samples,” Jowr. Appl. Phys., vol. 23, no. 7; 1952. 

8S, O. Rice, Bell Sys. Tech. Jour., vol. 23, p. 73. 1945, 
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second-order probability distribution for the envelope 
R of the filter output is 


Bes ue Rk, 
PUEOIIS) sta Ue Ty 3} 
ea se R3 
‘exp — 1 te 1 — Re (GAD) 


Here I,(x) is the modified Bessel Function of zero order 
and 


iP i_ w(fi)w( fe) cos 2n(f, — fo)t df, dfs 


{ [ ar} 


= W(r) = (72) 


a= | wif) af, (73) 
0 
when w(f) is the power spectrum of the noise at the 
output of the filter. 
The corresponding first order distribution has the very 
simple form (Rayleigh distribution) 


p(k) = a exp { — 2 LE 


We now suppose that an “instantaneous” square law 
envelope detector is connected to the output of our 
narrow band filter. Such a detector would have a time 
constant, long compared to the reciprocal of the filter 
midband frequency, but short compared to the reciprocal 
of the filter bandwidth. These are conflicting requirements, 
but are usually closely approximated in practice. Then 
denoting the outputs at times ¢, , ¢, by x, and 2, re- 
spectively, we may write 


(74) 


Rites as (75) 
Qo R; 
From (71), we find immediately 
eo Cite fig ie vee 5} 1) UN ain A Ti Le 
pla, vee) a do! Io eat Ae (D4 0) ; Pa a i) 
(76) 


and from (74), we have for the first-order distribution 


p(t) = 5 exp i= as (77) 
We now make use of the identity”* 
(1 — 7 exp { -(@ +9) tps? autl 
= LtL@LW, (8) 
when L,,(x) is the Laguerre Polynomial defined”’ by 
£ we “= nile 2h): (79) 


24 G. N. Watson, “Generating functions of polynomials I,” Jour. 
London Math. Soc., vol. 8, p. 189; 1933. This is a special case of the 
more general result given by Watson. 

28 G. Szego, “Orthogonal polynomials, ” Amer. Math. Soc. Collo- 
quium Pub., vol. 23, p. 97; 1939. 
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Using (78) we may write (76) in the form 


1 1 efit “2 - Wn 1 192, 
(80) 
= 2\n vy v2 
p(x1)p(x2) 2 (uw) z4(2s).( 2) (81) 


where use has been made of (77). When it is recalled that 
the orthogonal property for the Laguerre Polynomials 
* 26 

is 


[cP LaCayEn@) bt = bn (82) 
0 
it is seen that (81) is already in standard form with 
£ 
One) ib (83) 
a, = (w)” 


showing that p(x, , x.) defined by (75) is in A. 


Hxample 3 


As a final example, we show that the second-order 
probability distribution for a sine wave of constant 
amplitude P is in A. It will be most convenient in this 
example to start from the characteristic function of the 
distribution. If we denote our sine wave by 


a(t) = p cos (wi + 4), (84) 
_ the second-order characteristic function defined as 
TOO se (85) 
has been shown by Rice’ to be 
g(u,v) = Jnofp Vu? + v? + 2uw cos wr}. (86) 


It follows that the second order probability distribution 
p(x, , ®) is given by the Fourier transform, 


P(x, , 2) = al / Jo{pVu? + v? + 2w cos wr} 


Oe ee Le, (87) 


To make further progress, we use Neumann’s addition 
theorem,” namely 


Joi Vw + v” — 2uv cos $} 


foe) 


= Dy ea) AU) Jo) COs, (88) 
n=0 
where eo) Bimal (89) 
= 2 Te 1 2, 


Using (88) in (87) and reversing the order of summation 
and integration, we find 


26 Thed. 

278. O. Rice, “Mathematical analysis of random noise,” Bell Sys. 
Tech. Jour., vol. 24, p. 138; 1945. 

2GS, IN Watson, “Theory of Bessel Functions, ” Cambridge Dints 
versity Press, Cambridge, Eng., p. 358; 1922. 
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P(X, , %) = SS (—1)"e, cos nwr 
n=0 


1 ‘a —tUaty 1 ue e aie 
8 i ; J (puje iu i Gye iv. (90) 


We now use the result”® 


vile ea 5 tue 9, ks 1) oe Ax/p) wy ] 
oe i J ,(puye n= cp = eee 


(91) 
Here T,,(x) is the Tchebycheff Polynomial of the first kind 


defined by 


T(x). = cos {n cos x}. (92) 


Using (91), we see that (90) may be written 


le Ley ee 
pn a= a) A ae 
= ti\,, (x 
Se eat )0,(2) cosnwr, (93) 
n=0 
when i : 
ae = 
Pp Pp 
otherwise 
p(X, , X) = 0. 


The first-order probability distribution for the sine 
wave has the well-known form’ 


2 ree 


pa) = + (94) 


and the Tchebycheff Polynomial has the orthogonal 
property 


1 


2 fhe. é,f, AT Aa — 2) de =. 6, (95) 


so it follows that (93) may be written in the standard form 


pler 5 #2) = peaiiples) Lain On(t2), (96) 
where “d \ 
aa) = Ve r(2) te 
Q, = COs nwr 


Note that, from (92), we may write a, = T,,(a). 
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Predictive Coding—Part I 


PETER ELIASt 


Summary—Predictive coding is a procedure for transmitting 
messages which are sequences of magnitudes. In this coding method, 
the transmitter and the receiver store past message terms, and from 
them estimate the value of the next message term. The transmitter 
transmits, not the message term, but the difference between it and 
its predicted value. At the receiver this error term is added to the 
receiver prediction to reproduce the message term. This procedure 
is defined and messages, prediction, entropy, and ideal coding are 
discussed to provide a basis for Part Il, which will give the mathe- 
matical criterion for the best predictor for use in the predictive coding 
of particular messages, will give examples of such messages, and 
will show that the error term which is transmitted in predictive 
coding may always be coded efficiently. 


INTRODUCTION 


NWO MAJOR contributions have been made within 
‘Le past few years to the mathematical theory of 
communication. One of these is Wiener’s work 
on the prediction and filtering of random, stationary time 
series, and the other is Shannon’s work, defining the 
information content of a message which is such a time 
series, and relating this quantity to the bandwidth and 
time required for the transmission of the message.’ This 
paper makes use of the point of view suggested by Wiener’s 
work on prediction to attack a problem in Shannon’s 
field: prediction is used to make possible the efficient 
coding of a class of messages of considerable physical 
interest. 

Consider a message which is a time series, a function m; 
which is defined for all integer 7, positive or negative. 
Such a series might be derived from the sampling used in 
a pulse-code modulation system.” From a knowledge of 
the statistics of the set of messages to be transmitted, we 
may find a predictor which operates on all the past values 
of the function, m; with 7 less than 7, and produces a 
prediction p, of the value which m will next assume. 
Now consider the error e; , which is defined as the differ- 
ence between the message and its predicted value: 


Sf Di . (1) 


All of the information generated by the source in 
selecting the term m, is given just as well by e; ; the error 
term may be transmitted, and will enable the receiver to 
reconstruct the original message, for the portion of the 
message that is not transmitted, p; , may be considered 


+ Elec. Engrg. Dept. and Res. Lab. Elec., Mass. Inst. Tech., 
Cambridge, Mass. 

1 For historical remarks on the origin of modern information theory 
see C. E. Shannon and W. Weaver, ‘“‘The Mathematical Theory of 
Communication,”’ Univ. of Illinois Press, Urbana, Ill., p. 52 (foot- 
note) and p. 95 (footnote); 1949. 

2B. M. Oliver, J. R. Pierce, and C, E. Shannon, “The philosophy 
of PCM,” Proc. LR.E., vol. 36, pp. 1324-1331; November, 1948; 
also, W. R. Bennett, “Spectra of quantized signals,” Bell Sys. Tech. 
Jour., vol. 27, pp. 446-472; July, 1948. 


as information about the past of the message and not 
about its present; indeed, since p; is a quite determinate 
mathematical function, it contains no information at all 
by Shannon’s definition of this quantity.* 

The communications procedure which will be discussed 
is illustrated in Fig. 1. There is a message-generating 
source that feeds into a memory at the transmitter. The 
transmitter has a predictor, which operates on the past of 
the message as stored in the memory to produce an 
estimate of its future. The subtractor subtracts the 
prediction from the message term and produces an error 
term e,; , Which is applied as an input to the coder. The 
coder codes the error term, and this coded term is sent to 
the receiver. In the receiver the transmitting process is 
reversed. The receiver also has a memory and an identical 
predictor, and has predicted the same value p, for the 
message as did the predictor at the transmitter. When 
the coded correction term is received, it is decoded to 
reproduce the error term e; . This is added to the predicted 
value p; and the message term m; is reproduced. The 
message term is then presented to the observer at the 
receiver, and is also stored in the receiver memory to 
permit the prediction of the following values of the 
message. 


TRANSMITTER 
MESSAGE 
SOURCE 
Dass | suarnacrr | 


CODED 
SUBTRACTOR ERROR TERM 


RECEIVER 


ADDER MEMORY a PREDICTOR 


DECODER 


CODED 
ERROR TERM 


Fig. 1—Predicting coding and decoding procedure. 


This procedure is essentially a coding scheme, and will 
be called predictive coding. The memory, predictor, sub- 
tractor, and coder at the transmitter, and the memory, 
predictor, adder, and decoder at the receiver may be 
considered as complex coding and decoding devices. 
Predictive coding may then be compared with the ideal 
coding methods given by Shannon and Fano.* In general, 


: Se and, Weaver, op. cit., p. 31. 
‘Shannon and Weaver, op. cit., p. 30; also R. M. F Tech 
Rep. No. 65, Res. Lab. Elect., M.I.T., Cambridge, Mass.: 1949. = 
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predictive coding cannot take less channel space for the 
transmission of a message at a given rate than does an 
ideal coding scheme, and it will often take more. However, 
there is a large class of message-generating processes 
which are at present coded in a highly inefficient way, 
and for which the use of large codebook memories, such 
as are required for the ideal coding methods, is impractical. 
Time series which are obtained by sampling a smoothly 
varying function of time are examples in this class. For 
many such processes predictive coding can give an 
efficient code, using a reasonable amount of apparatus at 
the transmitter and the receiver. 

It should be noted that in the transmission scheme of 
Fig. 1 errors accumulate. That is, any noise which is 
introduced after the transmitter memory, or at the 
receiver, or in transmission, will be perpetuated as an 
error in all future values of the message, as will any 

discrepancy between the operation of the two memories, 
or the two predictors. This means that eventually errors 
will accumulate to such an extent that the message will 
disappear in the noise. If, therefore, continuous messages, 
1.e., time series each member of which is selected from a 
continuum of magnitudes, are to be transmitted, it will be 
necessary periodically to clear the memories of both the 
receiver and the transmitter and start afresh. This is 
undesirable, since after each such clearing there will be 
no remembered values on which to base a prediction, and 
more information transmission will be required for a 
period following each such clearing, until enough re- 
membered values have accumulated to permit good 
prediction once more. 

A more satisfactory alternative is the use of some pulse- 
code transmission system in which only quantized magni- 
tudes of input are accepted. Such a system may be made 
virtually error-free.” A system of this kind has the further 
advantage that the only very reliable memory units now 
available or in immediate prospect are of a quantized 
nature, most of them being capable only of storing binary 
digits. The use of a quantized system requires that the 
predicted values be selected from the permissible quantized 
set of message values. Strictly interpreted, this severely 
limits the permissible predictors; if by a choice of scale 
the permissible quantized levels are made equal to the 
integers, then the restriction on p(m,_; --- m;_,) 18 that 
it take integer values for all sets of integer arguments. 
Actually the ordinary extrapolation formulas have this 
property, and may be used as predictors. But it is not 
necessary to limit the choice of predictors so severely. 
The problem may be evaded by using any function as a 
predictor and computing its value to a predetermined 
number of places by digital computing techniques, the 
prediction then being taken to be the function rounded 
off to the nearest integer. If the predictor as originally 
computed was optimum in some well-defined sense, then 
the rounded predictor will presumably be less good in that 
sense, but in cases where predictive coding may be 
expected to be useful the difference will usually be small. 


5 Oliver, Pierce, and Shannon, loc. cit. 
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It is necessary to define precisely what is meant by an 
optimum predictor for use in predictive coding—i.e., to 
define some quantity, which depends upon the choice of 
the predictor, and define as optimum a predictor which 
minimizes this quantity. Wiener’s work uses as a criterion 
the minimization of the mean square error term e’. 
Wiener has pointed out that other criteria are possible, 
but that the mathematical work is made simpler by the 
mean square choice.” Minimizing the mean square error 
corresponds to minimizing the power of the error term, 
and if no further coding is to be done, this is a reasonable 
criterion for predictive coding purposes. However, in the 
system illustrated in Fig. 1, the error term is coded before 
it is transmitted, and its power may be radically altered 
in the coding process. What we are really interested in 
minimizing is the channel space which the system will 
require for the transmission of the error term. This leads 
to the following criterion which will be justified in Part IT 
of this paper: That predictor is best which leads to an 
average error-term distribution having minimum entropy. 

The coder of Fig. 1 also requires some consideration. 
Predictive coding eliminates the codebook requirement 
by using prediction. To take advantage of the resultant 
savings in equipment, it is necessary to show that the 
coder itself will not require a large codebook. This reduces 
to the problem of showing that a message whose terms 
are assumed independent of one another may always be 
coded efficiently by a process with a small memory require- 
ment. It will be shown that this is true. It is necessary to 
use two kinds of coding processes: one for cases in which 
the entropy of the distribution from which the successive 
terms are chosen is large compared to unity, and another 
for cases in which the entropy is small compared to unity. 

The following sections of the present paper are devoted 
to a discussion of messages, prediction, entropy, and ideal 
coding. Part II will discuss the predictor criterion given 
above, the classes of messages for which a predictor that 
is optimum by this criterion may be found, and other 
classes of messages for which predictive coding may be of 
use. Mathematically defined examples of message- 
generating processes which belong to these classes will be 
given, and the problem of coding the error term so as to 
take advantage of the minimal entropy of its average 
distribution will be examined. 


CHARACTERIZATION OF MESSAGES 


A necessary preliminary to a discussion of messages is a 
precise definition of what “message” is taken to mean.’ 
Since a communication system is designed to transmit. 
many messages, what is actually of interest is the 


6 N. Wiener, ‘The Extrapolation, Interpolation and Smoothing: 
of Stationary Time Series with Engineering Applications,”’ published 
in 1942 as an NDRC report, and in 1949 as a book, by the Mass. Inst. 
Tech. Press, Cambridge, Mass., and John Wiley & Sons, Inc., New 
York, N. Y., especially p. 138. 

7Such definitions are given by Wiener, zbzd., and Wiener,. 
“Cybernetics,’’ Mass. Inst. Tech. Press, and John Wiley & Sons, 
Inc., 1948; also by Shannon and Weaver, loc. cit. Our discussion 
starts with a definition like Wiener’s and ends with one like Shan- 
non’s. 
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characterization of the ensemble from which the trans- 
mitted messages are chosen, or the stochastic process by 
which they are generated. As a preliminary definition, 
we may say that a message is a single-valued real function 
of time, chosen from an ensemble of such functions. It 
will be denoted by m(a, ft), where a is a real number 
between zero and one which labels the particular message 
chosen from the ensemble, and m(a, ¢) is defined, for 
each such a, for all values of ¢ from —o to o. This 
definition must be restricted in several respects, in part 
to take into account the physical requirements of trans- 
mitting systems and in part for mathematical con- 
venience. 

First, it is assumed that the ensemble from which the 
messages are chosen is ergodic. This means that any one 
message of the ensemble, except for a set whose measure 
in a is zero, is typical of the ensemble in the following 
sense: let Q(a) be the probability distribution of the param- 
eter of distribution a. Then with probability one, for 
any function f[m/(a, t)] and almost any a, , 


lin se [Soma Q1ae =f flm(a, D1 dQ@. 2) 


Toe 4 


L.e., any function of m has the same average value when 
averaged over time as a function of a single message, as 
when averaged over the ensemble of all possible messages. 
We can thus find out all possible statistical information 
about the ensemble by observing a single message over 
its entire history. The ergodic requirement implies that 
the ensemble is stationary: i.e., that the statistics do not 
change with time. Its practical importance is that it 
permits us to speak indifferently of the message or the 
ensemble, and makes it unnecessary to specify the sense 
in which we speak of an average. In particular, it permits 
the substitution of measurable time averages for ex- 
perimentally awkward ensemble averages. 

Second, it is assumed that the average square of the 
message [in either sense of (2)] is finite. The message will 
be represented in physical systems by a voltage or a 
current, or the displacement of a membrane, or the 
pressure in a gas, or by several such physical variables, as 
it proceeds from its origin to its destination. All of these 
representations require power; in particular, representa- 
tion as a voltage or a current between two points separated 
by a fixed impedance, which is a necessary intermediate 
representation in any presently used electrical communi- 
cation method, requires a power proportional to the 
square of the message. Since only a finite amount of 
power may be supplied to a physical transmitter, it is 
obviously required that the average message power be 
bounded. 

Third, it is assumed that the spectrum of the message 
vanishes for frequencies greater than some fixed frequency 
fo . This will not in general be true for the radio-frequency 
spectrum of the messages as they are generated by a 
source, and it has been shown that a function with an 
infinitely extended spectrum cannot be reduced to a 
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function with a spectrum of finite range by any physically 
realizable filter; the transfer characteristic of a filter can 
be zero only for a set of frequencies of total measure 
zero. However, this is no practical problem. For since 
the message has a finite total power distributed over the 
spectrum, there will always be an f, so high that a 
negligible fraction of the total power will be located 
beyond it in the power spectrum. 

The reason for this assumption is that, as Shannon has 
pointed out, any function of time that is band-limited 
may be replaced by a time series, which gives the values 
of the function at times separated by an interval 1/2f, .” 
For any band-limited function we have the following 
identity: 


m() = > m(i/ago sina ehat =O} (3) 


The values of the function at the sampling points 
t = 7/2f) , which are the coefficients of this series, thus 
completely determine the function. If the function is not 
initially band-limited, the expansion will give a function 
which passes through the same values at the sampling 
points, but which is band-limited. As we assume band- 
limited messages, for our purpose the series and the 
function are equivalent, and since the series is easier to 
deal with in the sequel, it is desirable to change the 
definition of the message. Henceforth the message will be 
defined as the series of coefficients in the expansion (3). 
By choice of the unit of time, the sampling interval is 
made unity, and the message is then m;(a), defined for all 
(positive and negative) integer values of the index 7. 

A message is thus a time series drawn from an ergodic 
ensemble of such series, and each term in any one message 
is drawn from a probability distribution whose form is 
determined by the preceding terms of that message. 
For the reasons indicated in the first section, we will be 
interested primarily in quantized messages, for which this 
probability distribution will be discrete. However, it will 
at times be more convenient in the analysis and the 
examples to deal with continuous distributions, it being 
understood that quantization will ultimately be used. In 
the discrete case, the message term m; will be selected 
from a discrete probability distribution M, , where 
M,(m;-1 m;-; +++) is the conditional distribution 
giving the probability that, for a particular set of past 
values m;_; -+- m;_; ++: , the message term m, will take 
the integer value k. In the continuous case, the message 
term m, will be chosen from a continuous conditional dis- 
tribution M(m,; : m;_, --+ m;_; --+). Both of these dis- 
tributions are dependent on the set of values of the pre- 
ceding message terms m;_; +--+ m;_; «++ , but are of course 
independent of the value of the index 7, by the stationary 
nature of the ensemble. 


8 Wiener, “The Extrapolation, Interpolation and Smoothing 
of Stationary Time Series with Engineering Applications,’ NDRC 
Report, Mass. Inst. Tech. Press, Cambridge, Mass., p. 37: 1942. 

°C. KE, Shannon, “Communication in the presence of noise,” Proc 
LR.E., vol. 37, pp. 10-21; January, 1949. i ; 
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Stochastic processes of this sort are known as Markoff 


processes and have an extensive mathematical literature.”° 


An nth order Markoff process is one in which the dis- 


tribution from which each term is chosen depends on the 


set of values of the n preceding terms only; a process in 


/which each term is chosen from a single unconditional 


probability distribution may be called a Markoff process 
of order zero. It should be noted that, while any Markoff 
process yielding a message with a finite second moment 
is included in this definition, we will expect most of the 
messages to be Markoff processes of a rather special kind. 
The messages have been derived by the time-sampling of 
a continuously varying physical quantity. The sampling 
rate must be high enough so that the sampling does not 


suppress significant variations in the message—i.e., the 


fo must be above the bulk of the spectral power of the 
message. Now for most such messages, the average rate of 


: variation with time is much lower than the highest rate 
that the system must be capable of transmitting. Con- 


sequently, it is to be expected that on the average, suc- 
cessive message values will be near to one another. This 
means, in particular, that in the discrete case the index k 
is not just an arbitrary labeling of a particular symbol— 
as it is, for example, in Shannon’s finite-order Markoff 
approximations to English’’—but may be expected to give 
a genuine metric: message values with indexes near to 
one another may be expected to have probabilities near 
to one another, and the conditional distributions mentioned 


above may be expected to be unimodal. This is not a 


restriction on what kinds of series will be considered to be 
messages, but is rather a specification of the class of 
messages for which predictive coding may be expected to 


be of use, as will be discussed in detail in Part II of this 


paper. 

For a message ensemble for which the conditional dis- 
tributions are not given a priori, it 1s necessary to de- 
termine them by the observation of a number of messages, 
or of a single message for a long time. It is obviously 
impossible to do this on the assumption that the distri- 
bution from which a particular message term is chosen 
depends on the infinite set of past message values. What 
can, in fact, be measured are the zeroth order approxi- 
mation, in which each term is treated as if it were drawn 
from the same distribution, giving M(m;), an unconditional 
distribution; the first order conditional distribution 
M(m; : m;-:), and so on to the nth order conditional 
distribution for some finite n. A communications system 
which is designed to transmit this approximation will be 
inefficient: the approximating process itself would generate 
messages with a greater information content than the 
messages which are actually being transmitted, and a 
system designed for the approximation will waste time 
or power or bandwidth when transmitting the real message. 
This will be discussed more fully later. 


10 Shannon and Weaver, op. cit., p. 15; also, M. Frechet, cited 
there, and P. Levy, ‘‘Processus Stochastique et Mouvement Brown- 
ien,’’ Gauthier-Villars, 1948, which give further references. 

1 Shannon and Weaver, op. cit., pp. 9-15. 
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PREDICTION 


Norbert Wiener has developed a very general method 
for finding the linear predictor for a given ensemble of 
messages which minimizes the root mean square error of 
prediction. His method was developed for the difficult 
case of nonband-limited messages, i.e., continuous 
functions of time which cannot be reduced to time series. 
However, he has also solved the much simpler problem of 
the prediction of time series, such as the messages which 
were defined above. The details of this work are thoroughly 
covered in the literature,” and this section will merely 
define some terms, note some results, and discuss the 
prediction problem from a point of view which is weighted 
towards probability considerations and not. towards 
Fourier transform considerations. 

From a time series, a linear prediction p; of the value 
of a message term m, is a linear combination of the previous 
message values 


Diel= ys a;M;-; . 
7=1 


The error e, is defined as 
C5 —= fDi. > MNS © 


The predictor itself may be considered to be the set of 
coefficients a; . The best linear predictor, in the rms sense, 
is the set of coefficients which, on the average, minimizes 
e’. Wiener has shown that this predictor is determined, not 
by the message ensemble directly, but by the auto- 
correlation function of the ensemble. In general, there will 
be many ensembles with the same autocorrelation function, 
and the same linear predictor will be the best in the rms 
sense for all of them. 

The autocorrelation function for a time series is defined 


by 


1 
IN 1 


N 
Dai ae 


i=—N 


C= lim 


No 


Devices for rapidly obtaining approximate autocorrelation 
functions have been constructed.'* These devices accept 
the message directly as an input, and graph or tabulate 
the function. By the use of such devices, or by a statistical 
examination of the message, or in some cases by an @ 
priort knowledge of the message-generating process, it is 
possible to determine the autocorrelation function. The 
best linear predictor in the rms sense may then be de- 
termined. But it should be noted that there may be non- 
linear predictors which are very much better. 

Indeed, given a complete knowledge of the stochastic 
definition of the message, i.e., a complete knowledge of 


2 Wiener, op. cit. Also H. W. Bode and C. E. Shannon, “A 
simplified derivation of linear least square smoothing and prediction 
theory,” Proc. IRE, vol. 38, pp. 417-425; April, 1950. 

%T, P. Cheatham, Jr., Tech. Rep. No. 122, Res. Lab. Elect., 
M. I. T. (to be published). See also, Y. W. Lee, T. P. Cheatham, Jr., 
and J. B. Wiesner, ““The Application of Correlation Functions in the 
Detection of Small Signals in Noise,’’ Tech. Rep. No. 141, Res. Lab. 
Elect., M. I. T.; 1949. 
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the conditional probability distributions W(m; :m;_, --: 
i-; -°:) or M,(m,_1 +++ m,_; ---) the best rms predictor, 
with no restriction as to linearity, is directly available. 
Obviously the best rms predictor for a message term m, 


m 


defined in this way is the mean of the distribution from 
which it which is determined by the past 
message history: 1.e., the best rms predictor, p*, is 


S55 kM,(m:-1 -°: 


k=—-©@ 


is chosen, 


Mi; o -) 
or 


--) dm, 


/ HONING: 2 Ux 2° o Tia 


in the discrete and continuous cases respectively. For the 
mean of a distribution is that point about which its 
second moment is a minimum. Of course, the mean need 
not be a linear function of the past message values. How- 
ever, it is some determinate function of these values unless 
the message values are completely uncorrelated—i.e., 
unless the Markoff process is of order zero. In this case, 
it is just the constant which is the mean of the zero-order 
distribution. We therefore have as the unconditionally 
best rms predictor the function p*(m,;_, +++ m,_; ++). 

From this same general statistical viewpoint the best 
predictor on a mean-absolute error basis is the prediction 
of the median of the conditional distribution, since the 
median is that point about which the first absolute 
moment is a minimum. Like the mean, the median is 
defined by the conditional distribution M as a function 
of the past history of the message. This definition may 
not be unique: if there is a region of zero probability 
density between the two halves of a probability distri- 
bution, any point in the region is a median. However, the 
definition may be made unique by selecting a point within 
this range, for those sets of past message values for which 
the ambiguity arises. We will denote the best predictor 
in the mean-absolute sense by p**, it being understood 
that the definition has been made unique in some suitable 
way if the ensemble is such as to require this. 

Finally, it may be desired to predict in such a way that 
in the discrete case, the probability of no error is a maxi- 
mum, and in the continuous case the probability density 
has the maximum possible value at zero error. This 
requires modal prediction. The mode of the conditional 
distribution will not be unique if there are several equal 
probabilities which are each larger than any other prob- 
ability in the discrete case, or if the continuous distribution 
attains its maximum value at more than one point. The 
difficulty may again be removed by a suitable choice, and 
p*** will signify the best modal predictor. 

In any of these cases, and indeed for any other prediction 
criterion which yields a determinate value of the prediction 
as a function of the past history of the message, the error 
term e,; is drawn from a distribution H(e,; :m;_,-+-m;_; +++) 
or Ey(mz4 --: Mi; -) which is of exactly the same 
form as the original distribution of the message term, but 
which has been shifted along the axis by the amount of 
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the prediction. If it is desired to limit predictions to the 
possible quantized values of a discrete probability dis- 
tribution, it is only necessary to make p** and p*** unique 
in a way which does this in the cases of ambiguity; where 
the median and mode are uniquely defined, they will 
always coincide with one of the possible values of the 
message. For rms prediction it is necessary to take the 
quantized value that is nearest to the computed mean 
of the distribution as the value of p*. 
As an example of a predictable function, consider 


Mant ine 7 exp [—(m; — am;_,)’/20°]. (4) 


o aT 
The unconditional distribution of m; may be found by 
using the reproductive property of the normal distribution. 
M(m;) will be normal, with a standard deviation o’, and 
am,_, will have a normal distribution with standard 


deviation ao’ : then, 
¢ 00° =o s og = a (5) 
—a 
and 
== 1 2 
M(m; = ——= exp [—m‘/20’”]. (6) 
o! V2 


The zero-order approximation to this first-order Markoff 
process has, then, a message term distribution of the 
same form as the original conditional distribution, but a 


2 


standard deviation which is larger by a factor 1/ wt Oe 
By our definition in a previous section the process will 
generate messages only if a < 1: otherwise the standard 
deviation will be infinite, and the message will require 
infinite power for transmission. A more general example 
in complete analogy to (4) is: 


M( 


Mga, °° ) 


agrel-lo-Esa)/2} 0 


Wiener’s prediction procedure is designed for functions 
of the form (7), in which each term of the time series is 
drawn from a normal distribution with constant o, with a 
mean which is a linear combination of past values, the 
permissible combinations being limited by the requirement 
that the resultant average distribution have a finite 
second moment. The linear combination of past values 
which is the mean of the conditional distribution is also 
the best linear rms predictor, and is indeed the best rms 
predictor p*, as noted above. Wiener’s method is then a 
procedure for finding this linear combination in terms of 
the autocorrelation function of the message. 

The combination of past terms in the exponent may be 
rewritten as a sum of differences, less a constant times the 
message value m; . The stochastic function determined by 
the conditional distribution will then be as approximation 
to the solution of the difference equation obtained by 
setting the exponent in (7) equal to zero. In the limit 
¢ — 0, the stochastic function will become precisely the 
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function which is a.solution to this equation, as determined 
by the set of past message values (initial conditions): as o 
grows, the function will wander about in the neighborhood 
of this solution, diverging from it more and more as 7 
increases. In (4) above, the equation obtained is just 
m,; — am;_, = 0, and the solution, m; = am;_, , gives a 
geometric approach to the origin. 

In the case of continuous functions of time, taking 
appropriate limits gives a normal distribution about a 
linear function of the past which may include integral or 
differential operators on the past. The bulk of Wiener’s 
analysis is devoted to this case. Although the method was 
designed with functions like (7) in mind, it is clearly not 
limited to them. In the case of time series it is possible to 
use a distribution which is not normal, with a standard 
deviation (or other parameter or parameters) which is not 
constant, but is also determined by the past values of the 
message. So long as the mean of the distribution is still a 
linear combination of past values, the predictor derived 
from the autocorrelation function will still give the best 
rms predictor. If the mean is a nonlinear function of the 
past values, the predictor obtained from the autocorrela- 
tion function will be the best linear approximation to this 
nonlinear function in the rms sense. 

Where the best predictor is indeed linear, or is well 
approximated by a linear combination of past values, the 
great practical superiority of Wiener’s method over the 
use of the conditional distribution should be clear. For in 
this method only the autocorrelation function, a function 
of a single variable, needs to be measured; the predictor 
can then be computed no matter what the order of the 
Markoff process may be. Using the conditional probability 
distribution directly, an nth order Markoff process will 
require the observational determination of a function of 
n + 1 variables. This becomes a task of fantastic propor- 
tions when n is as large as four or five: it is practical for 
small n only for a quantized system with very few possible 
quantized levels. 

The direct use of the conditional distribution may, 
however, be quite valuable if the best rms predictor is a 
highly nonlinear function of only a few past values, 
particularly in a quantized system. Nonlinearity is no 
more difficult to treat than is the linear case as far as 
analysis by this method is concerned. For the synthesis 
problem the lack of suitable nonlinear elements for the 
physical construction of nonlinear operators on the past 
is confined to the case of continuous functions of time; in 
the case of time series with quantized terms, digital com- 
puter techniques can provide any desired nonlinear func- 
tion of any number of variables—at, of course, an expense 
in equipment which may become very large for large n. 

When the conditional distribution always has a point of 
symmetry, we may note that the best rms predictor p* is 
equal to the best mean absolute predictor p**. If the 
distribution is also always unimodal, then the best modal 
predictor p*** will also be the same as p*. In particular, 
this will be the case for the examples (4) and (7), but it 
does not, of course, depend on the linearity of the predictor. 
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Entropy, AVERAGING, AND IDEAL CopDING 


The entropy H of a probability distribution M has 
been defined as’* 


H = — >> M, log M, 
k=—o@ 
and 
ji -| M(m,) log M(m,) dm; (8) 


in the discrete and continuous cases, respectively. The 
entropy of a probability distribution may be used as a 
measure of the information content of a symbol or message 
value m,; chosen from this distribution. The choice of the 
logarithmic base corresponds to the choice of a unit of 
entropy: when logarithms are taken to the base two, as is 
convenient in many discrete cases, the unit of entropy is 
the “bit,” a contraction for binary digit, since in a two- 
symbol system with the two symbols equiprobable, the 
entropy per symbol is one bit for this choice of base. In 
the continuous case computations are often made simpler 
by the use of natural logarithms. The resultant unit of 
entropy is called by Shannon the natural unit. We have 
one natural unit = log,e bits. 

Wiener, Shannon, and Fano™* give a number of reasons 
for the use of this function as a measure of information 
per symbol, and the arguments are plausible and satis- 
fying, but as Shannon remarks, the ultimate justification 
of the definition is in the implications and applications of 
entropy as a measure of information.’ For the analysis of 
communications systems, the definition is completely 
justified by theorems which prove that it is possible to 
code any message with entropy H bits per symbol in a 
binary code which uses an average of H + e binary digits 
per message symbol, where « is a positive quantity which 
may be made as small as desired, and by equivalent 
theorems in the case of the discrete channel with noise 
—1.e., where there is a finite probability that a symbol 


14 This is the definition given by Shannon (Shannon and Weaver, 
op. cit.) and Fano (R. M. Fano, ‘“The Transmission of Information’, 
Tech. Rep. No. 65, Res. Lab. Elec., M. I. T.; 1949) Wienes (‘‘Cyber- 
netics’, op. cit., p. 76) gives a definition with the opposite sign. There 
is no real conflict here, however, for Wiener is talking about a differ- 
ent measure. Wiener asks, how much information we are given about 
a message term, whose exact value will never be known, when we are 
given the probability distribution from which it is chosen. The an- 
swer is that we know a good deal when the distribution is narrow, and 
very little when the distribution is broad. Correspondingly, entropy 
as Wiener defines it has a large positive value for very narrow dis- 
tributions and a large negative value for very broad distributions. 
This measure is useful in determining how much information has been 
transmitted when a message term which is contaminated by noise 
with a known distribution is received; we can use Bayes’ theorem 
and find the probability distribution of the original message, and 
measure information transmitted by measuring the entropy of this 
distribution. Shannon, on the other hand, asks how much informa- 
tion is transmitted by the precise transmission of a message symbol, 
when we know a priori the probability distribution from which it was 
selected. In this case, if the distribution is very narrow, the message 
term tells us very little when it arrives; we knew what it would be 
before we received it. If the distribution is broad, however, then the 
arrival of the term tells us a good deal. This requires the use of the 
opposite sign for entropy. Shannon’s definition will be used through 
this paper; it is the more appropriate one for the kind of problem 
with which we are concerned. 

15 Shannon and Weaver, op. cit., p. 19. 
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transmitted at one quantized level will be received at a 
different level, and in the case of the continuous channel 
with noise—in which the message term is chosen from a 
continuous distribution, and is received mixed with noise, 
so that each received term is the sum of a signal term and a 
noise term, and reception is always approximate. 

For messages as defined in a previous section, we have, 
in general, that the entropy of the distribution from which 
any single message term is chosen is a function of the 
message history: in both the continuous and discrete cases 
we are concerned with conditional distributions, whose 
form depends on the set of values of the terms m;_, --- 
m;—; +: which prcede the message term m, whose entropy 
is defined in (8). For such cases—i.e., Markoff processes of 
order one or greater—the entropy is defined in terms of the 
probability, not of each message term, but of a sequence of 
N message terms, the limit being taken as N approaches 
infinity. Following Shannon,'° we define Gy in the discrete 
and the continuous cases as 


Gy = —(1/N) iP oe i Mim, , +++ msn) 
‘log M(m; , --: m,n) dm, +--+ dm;_y 
= —(1/N) eh pee Dh, Mang): mop) 
-log Mim; , === men). (9) 


Then the entropy per symbol of the process is defined as 


H = limGy. (10) 


N-o 


The distribution M(m,; , --+ m;—y) in (9) is not a con- 
ditional but a joint distribution: the distribution which 
determines the probability of getting a given set of NV 
values for the N + 1 message terms m;_y to m; . Now the 
joint probability distribution of order N + 1 is related to 
the conditional probability distribution and the joint 
distribution of order N by 


M(m; erie m;—n) 


=(Mimp omg my) Mong pees (11) 


Min). 
Using the relation (11) in the expression (9), for a 


message generating process which is a Markoff process of 
finite order k, and taking the limit (10), we have 


rect af ME ange ye shige 
af M(m; 2 mia ++ 


‘log M(m, 


Mx) amiss ee dm;_:. 


Mx) 


LOT) seeped 


Nb) am.) (12) 


with a similar relation for the discrete case, in which the 
integrals are replaced by sums. In words, what (12) states 


16Shannon and Weaver, op. cit., p. 25. 
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is that the entropy for the process as a whole is just the 
average over-all past histories of the entropy of the 
conditional distribution of order k which defines the 
process: the information content per symbol of a message 
generated by such a stochastic process is the average of 
the entropies of the distribution from which the successive 
message terms are chosen. 

It was noted that only a finite order Markoff process 
can, in general, be used as a model of a message source, 
and that, in general, the use of such an approximation is 
inefficient. We may now state this more exactly. If a kth 
order Markoff process is approximated by a process of 
order less than k, then the entropy of the approximating 
process will be greater than or equal to the entropy of the 
original process, with the equality holding only if the 
original process is actually of order less than k: 1.e., only 
if the kth order conditional distribution can be expressed 
in terms of conditional distributions of lower order. The 
result holds also for suitably convergent processes of 
infinite order. It is a direct consequence of the following 
more general theorem. 


Averaging Theorem I 


Let P(x: y) be a probability density distribution of x, 
for each value of the parameter y: 1.e., for all y, 


i Ps dc ele 


and 
Paeeyea0 


for all x and y. Let Q(y) be a probability density distribu- 
tion of y: 


[ew ay =1 


Qly) = 0. 
Let R(x) be the distribution P(x: y) averaged over the 
parameter y, and let H’ be its entropy: 


RO) = [ QW)Ple: dy 


7 ee / Ra) lop Rear (13) 
Let H(y) be the entropy of the distribution P(x: y) as a 
function of the parameter y, and let H be its average 
value: 


AY) = -f PG: y) log Per y) dz 


H 


[awn ay. 


Then we always have H’ > H, and the equality holds only 
when the y dependence of P(x: y) is fictitious. In words, 
the’ entropy of the average distribution is always greater 
than the average of the entropy of the distribution. 
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_ The proof is given in the appendix.’’ The theorem 
‘remains true for discrete distributions, and the statement 
is unchanged except for the uniform substitution of the 
summation indexes 7 and 7 for the continuous variables x 
and y and the replacement of integrations by sums. By 
successive application of the proof it is also obvious that 
the result holds for a distribution which is a function of 
parameters y, to y, . The application to Markoff processes 
is direct, for a conditional distribution of order k — 1 may 
be expressed as an integral of the form R(x) in (13), 
where P(x: y) is the conditional distribution of order k and 
y is the term m,_,, . 

The theorem is also applicable to cases in which the 
dependence of the distribution on past history is not 
explicit. If the dependence of the distribution M(m;:m,_, 

- m;-,) on the set of past message values is through a 
dependence on one or several parameters (e.g., the mean 
and the standard deviation of a distribution are functions 
of the set of past message values but the distribution is 
always normal), the conclusion still holds: the entropy of 
the average distribution, averaged over the distribution 
of the parameters, is always greater than the average over 
the parameters of the entropy. This is illustrated by the 
example of (4). The average message term distribution of 
the process is a normal distribution with a standard 
deviation ¢//1 — a’, with an entropy which may easily 
be computed"* as 


Hy = log o V 2me = log (i/V1 — @), 


but each message term has a normal distribution with 
standard deviation, with entropy just 


H = logsV2re, 


which is thus the average entropy of the process as a whole. 
The difference between these two entropies may be made 
as large as we like by letting a approach one. 

A second averaging theorem which will be useful later 
deals with averages over a single distribution. 


_ Averaging Theorem II 
Let P(x) be a probability distribution with entropy H: 


Pia 0 for all x, 


[ P@ ie Sie 
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Let Q(z, y) be a weighting function: 
[ cena =f py =1, 


Qa, y) 2 0 


17 The content of this theorem is implied by the derivations leading 
up to Shannon’s fundamental theorem, Shannon and Weaver, op. 
cit., p. 28. However, the theorem can be stated and proved as a 
property: of entropy as a functional of a probability distribution, 
with no reference to sequences of message terms, and the proof is so 
straightforward and simple that the theorem deserves an independent 
statement. 

18 Shannon and Weaver, op. cit., p. 56. 
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Let R(x) be the averaged distribution with entropy H’: 


Ra) = [| PW,» dy 


H’ = —[ R(x) log R(@) ar. 
Then we always have H’ > H, and the equality holds only 
when the weighting function is a Dirac delta function. 

This theorem is given by Shannon.” It is also true in the 
discrete case: the equality then holds only if the average 
distribution R(x), or R; in the discrete case, is a mere 
permutation of the distribution P(x), or P; . 

At the beginning of this section it was stated that it is 
possible to code a message with entropy H bits per symbol 
by a coding method which uses H + e binary output 
symbols per input symbol, on an average. Such a coding 
scheme will be called an zdeal code. Shannon has given two 
such coding procedures, and Fano has given one which is 
quite similar to one of Shannon’s.”” We will call coding by 
means of Shannon’s second procedure, or by means of 
Fano’s method, Shannon-Fano coding. Both are procedures 
for giving short codes to common messages and long codes 
to rare messages. They are given in the references. We will 
here only note the important result. Coding a group of NV 
message terms at once, the average number A, of output 
binary symbols per input message symbol is bounded: 


Gye Se Ge Ne (14) 


Here Gy is the quantity defined in (9). As N increases. 
Gy approaches H, the true entropy of the process, so H, 
also approaches H. For a discrete process, an efficient code 
may be defined as one for which the ratio H/H, is near 
one. It is clear that there are two reasons why a Shannon- 
Fano code for small N may be inefficient: first, if Gy is 
small, the ratio Gy/H, may be small, if H, is near its 
upper bound in (14). Second, for small NV, Gy may be a 
poor approximation to H. 

It should be noted that it is not reasonable to define an 
efficiency measure for continuous distributions as a ratio 
of entropies. For a process which is ultimately to be 
quantized, the entropy of a continuous distribution does 
not approximate the entropy of the discrete distribution 
which is obtained by quantization, unless the scale of the 
variable in the continuous distribution is so chosen as to 
make the interval between quantized levels unity. Using 
a different choice of scale adds a constant to the entropy of 
the distribution, so that the ratio which defines efficiency 
is changed. For this reason, until a quantizing level spacing 
is chosen, it is possible to speak only of the differences 
between the entropies of continuous distributions, and 
not of their ratios. 


19 Shannon and Weaver, op. cit., p. 21, property 4 for the discrete 
case; p. 55, property 3 for the continuous case. 

20 Shannon and Weaver, op. cit., p. 29; Fano, op. cit. Shannon’s 
procedure is simpler to handle mathematically; Fano’s is perhaps 
somewhat simpler to grasp. Fano’s method is not quite completely 
determinate. In cases in which the two methods do not agree, Fano’s 
provides a more efficient code than Shannon’s. 
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APPENDIX 


Proof of Averaging Theorem I 
Expanding H’ by the definitions given, we have 


co 


mz / : dx ie dy Q(y)P(x: y) log [ dz Q(2)P(a: ap 


Adding and subtracting a term gives 


olen 
HS" i dx 1B dy Q(y)P(a: y) log P(x: y) 


} . dz Q(2)P(x: | 
+ | dy QP: toe I Poe) 


The quotient in the last integral cannot cause trouble, 
since the integrand as a whole approaches zero with 
P(«: y). Interchanging the order of integration in the first 
integral and using the definition of H gives 


we=H-[ def dyQwPe:y) 


is dz Q(z)P(a: z) 
EGA) 


-log (14) 


Changing the logarithmic base will multiply both sides 
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of (14) by the same constant, so we are free to use natural 
logarithms and measure entropy in natural units. Using 
the inequality logu < wu — 1 in the integral in (14) gives 


H’ > A — Ibs dx 7 dy Q(y)P(x: y) 


7 —o 


iE dz Q(z)P(a: 2) 
Pars) 


ape 
>utf af dy QypPCe: 9) 
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Integrating first with respect to 2, we have by the normali- 
zation requirements on P(x: y) and Q(y) that 


fA far sleae als Ville 


The equality can be realized only when logu = 1, or in 
this case when 

/ PG DOG) da Patan (15) 

For this to hold, P(a: y) must have no dependence on vari- 

able y, since y does not appear on the left of (15), Q.E.D. 


In the discrete case, the precise same proof holds when 
summations are uniformly substituted for integrations. 


Predictive Coding—Part I] 


Summary—In Part I predictive coding was defined and messages, 
prediction, entropy, and ideal coding were discussed. In the present 
paper the criterion to be used for predictors for the purpose of pre- 
dictive coding is defined: that predictor is optimum in the information 
theory (IT) sense which minimizes the entropy of the average error- 
term distribution. Ordered averages of distributions are defined and 
it is shown that if a predictor gives an ordered average error term 
distribution it will be a best IT predictor. Special classes of messages 
are considered for which a best IT predictor can easily be found, 
and some examples are given. 

The error terms which are transmitted in predictive coding are 
treated as if they were statistically independent. If this is indeed 
the case, or a good approximation, then it is still necessary to show 
that sequences of message terms which are statistically independent 
may always be coded efficiently, without impractically large memory 
requirements, in order to show that predictive coding may be prac- 
tical and efficient in such cases. This is done in the final section of 
this paper. 


DEFINITION OF INFORMATION-['HEORY CRITERION 
FOR PREDICTORS 


We have now a sufficient vocabulary and collection of 
results to define and discuss a criterion of prediction that 
is appropriate for the kind of communications scheme 


outlined in the Introduction. An obvious definition is: 
that predictor is best, in the sense of information theory, 
which requires the minimum channel space for the trans- 
mission of its error term. But this specification is not yet 
sufficient. It is necessary to define to some extent the way 
in which the error term is to be coded, in order to define a 
predictor uniquely for a given message-generating process. 

One procedure is to use Shannon-Fano coding for the 
transmission of the error term. This means that the 
predictor p(m;_; --- m;_; -:-) should be chosen to 
minimize the ensemble average of the entropy of the error 
distribution. The average of 


-| E(m; 5 UDE al eee MF +++) 


‘log E(m; : m;-1 --- m;_; +++) dm, (16) 
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is to be minimized, the averaging being done over the 


message term joint distribution M(m;_, --- m;_; ---).”" 
This procedure, however, does not lead to a determination 
of the predictor p at all. Indeed, the expressions (16) are 
quite independent of p. For as pointed out in Part I, the 
distribution of the error term e, is the same as the dis- 
tribution of the message term m,; except for a shift in the 
mean value, i.e., a translation of the function along the 
axis of the independent variable. Now the entropy of a 
distribution is quite independent of any such shift. It is a 
sort of measure of the dispersion of the distribution, and 
not of the location of its mean. So the number obtained 
by averaging (16) over the message term distribution 
will, for each predictor, have the same value—a value 
which is precisely the information content of the original 
message. It is clear that this must be so if entropy is to 
be a valid measure of information. The information con- 
tained in the error terms is precisely all of the information 
present in the original message, since either can be 
obtained unambigously from the other; and the trans- 
mission of either, by an ideal code, will take the same 
amount of channel space. 

There is, however, a significant information theory 
criterion for predictors. An efficient Shannon-Fano code 
for a highly predictable message-generating process 
requires, as discussed in Part I, that a large number N 
of terms be remembered and coded at one time, so that 
the average number of bits per symbol in the coded 
message Hy , which is nearly equal to Gy , will be near 
the true entropy of the process H. This imposes two 
requirements on the coding process. 

1. There must be an active memory which remembers 
the past NV message terms. 

_ 2. There must be a translator, or codebook memory, 
which remembers the codes for all possible groups of N 
message terms. If each term is chosen from an M-ary 
discrete distribution, there must be N” entries in the 
codebook. 

_ For the transmission of a written message, these re- 
quirements are not too difficult to meet. The message 
itself, available at the transmitter, meets the first, and a 
codebook of the sort used for the commerical cable codes 
meets the second to a degree, with not too much loss of 
time in the coding process. But for an automatic device 
transmitting a rapidly occurring message, with sampling 
intervals of the order of microseconds or less, requirement 
2 becomes highly impractical for many messages in which 
M and N may both be larger than ten. 

The usual solution to this problem is to ignore the 
correlations between successive message terms, and to 
code the message as if it were a zero-order Markoff process, 
in which each message term m; is chosen from the average 
message term distribution M(m,). In this treatment the 
channel space required for the transmission of the message 


% Although the discussion will continue to include both the dis- 
crete and the continuous cases, the reader is reminded that the con- 
tinuous distributions are used for mathematical convenience only, 
and ultimate quantization is always implied. Therefore the trans- 
mitted message is always discrete, and the Shannon-Fano coding 
method cited in Part I may always be employed. 
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is increased, as is shown in Part I, and is determined by 
the entropy of the average message term distribution, 
rather than by the true entropy of the process, which is 
the average entropy of the message term distribution. 
There is then no active memory requirement, and the 
codebook requirement of M codes for the M possible 
message levels is manageable in size, and may be realized 
by such devices as the binary coding tube.” 

Predictive coding may be used as an intermediate step 
between the two limits of ideal and zero-order coding. As 
in zero-order coding, the correlation between successive 
terms of the transmitted message is ignored; but the 
transmitted message is now the error term and not the 
original message. The channel capacity required by this 
procedure is then equal to the entropy of the average 
error term distribution, if we can find an efficient method 
of coding the error terms which does not require a large 
codebook. Such a coding method will be called non- 
mnemonic. It will be shown that this can be done. 

These considerations suggest the following definition. 
Let # be the average error distribution corresponding to a 
particular predictor p: 


Ee; ,p) = ie see [ Be. Lonaeg aie en eee) 
-M(mi-3 +++ ms; «+°) amy +++ dm; 
Aga Sn Seen 
Mm.) a ea (17) 


where M is the joint distribution of the prior message 
values as before. Then: 


Definition 

The best predictor, in the information theory sense, is 
that function p(m;_, --: m;_, +++) which minimizes the 
entropy of the averaged error distribution, E(e; , p) or 
E,(p). A 

The entropy of His by no means independent of the 
predictor p; a number of different distributions are being 
averaged, and in simple cases it is obvious that, if the 
distributions are narrow, the resultant curve will be broad 
if the means of the averaged distributions are widely 
dispersed, and narrow if they are near to one another. 
Entropy is a functional of a distribution which is large 
when the distribution is broad and small when it is narrow, 
and it is now reasonable to ask how to choose the means of 
the otherwise determined distributions which are being 
averaged, so as to minimize the entropy of the resultant 
average distribution. 

There are several general points which may be noted 
about this criterion of prediction. 

1. The addition of a constant to a best IT (Information 
Theory) predictor will give a best IT predictor. The 


2R. W. Sears, “Electron beam tube for pulse code modulation,” 
Bell Sys. Tech. Jour., vol. 27, pp. 44-57; January, 1948. 
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predictor is determined by this criterion only up to an 
additive constant. For, if the means of all of the distri- 
butions being averaged are shifted by the same amount, 
the resultant will have its mean shifted by an equal 
amount, but will not be otherwise changed, and the 
entropy is invariant under such shifts. 

2. Even if the additive constant has been determined 
by some additional constraint (such as the requirement 
that the mean of / be zero), the predictor may not be 
unique. An example will appear later. Here we only note 
that, while this will be rare in practical cases, there will 
be many cases in which there is a large class of predictors 
with very nearly the same entropy. 

3. The transmission method presented in Part I, using 
a predictor which satisfies this criterion, will not in general 
be ideal. It will be ideal in some important special cases, 
and it will be nearly ideal, and much more efficient than 
zero-order coding, in cases in which the error distribution 
is narrow for any past history but has a mean which 
changes widely as the recent history changes. In such 
eases the IT criterion will lead to predictors which are 
close to the best rms or other moment criterion predictors. 

From a practical point of view, this method is much 
more feasible in highly predictable cases than is the 
Shannon-Fano coding. As will be shown in an example, 
the active memory requirement may be much smaller; far 
more important, the codebook required is always small. 
Coding methods will be considered later in detail. It turns 
out that the codebook memory will rarely have more than 
M entries, and that when it is necessary to code error 
terms in groups, non-mnemonic coding may always be 
used. 

Unfortunately no elegant and general solution for the 
best IT predictor has been found, and there is reason to 
believe that it will not be possible to find such a solution 
in which no restrictions are placed on the message other 
than those listed in Part I. However, it is possible to find 
the best IT predictor when suitable additional restrictions 
are imposed. We will consider several sets of restrictions, 
which seem likely to cover most of the cases in which a 
quantized message is obtained by sampling a smoothly- 
varying function of time. In some of these the best IT 
predictor is obtained explicitly, but for certain significant 
cases approximation methods must be used. 


ORDERED AVERAGES AND SPECIAL CASES 


Predictive coding involves the averaging together of a 
number of error term distribution. As shown in Part I, 
such an averaging in general increases the entropy. An 
ordered average is a kind of average which minimizes this 
increase. It is not always possible to obtain an ordered 
average by using predictive coding, but when it is possible 
to do so, the predictor used will be a best IT predictor. 
In any case, the entropy of the ordered average may be 
computed from the statistics of the message, if these are 
sufficiently well known, and it provides a lower bound to 
the savings in channel space which may be achieved by 
the use of predictive coding. It is thus useful in deciding 
whether or not it is worthwhile to investigage predictive 
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Consider a set of n discrete distributions, each having a 
total of m terms, and let M, represent the probability, in 
the jth of these distributions, of the occurrence of the kth 
symbol. For a particular set of averaging weights a; with 


a; = O 
2 a = i 
7=1 


the ordered average is defined by taking a, times the largest 
probability in the first distribution, a, times the largest 
probability in the second distribution, and so on. The 
weighted average of these largest probabilities is the first 
term, M, , in the ordered average. Using the same weights 
and the second largest probabilities in each distribution 
gives the ordered average term M, , and so on to the 
average of the smallest probabilities, which gives /,, . 

It can be shown that the entropy of the ordered average 
distribution is less than that of any other average formed 
from the same set of probability distributions with the 
same weights, but with the terms of one or more of the 
distributions not arranged in order of decreasing prob- 
ability.”’ There are, therefore, in general, four different 
entropies to be considered in the predictive coding of a 
message: H, , the entropy of the average message term 
distribution; H’, the entropy of the average error term 
distribution using the best IT predictor; H’, the entropy 
of the ordered average error term distribution; and H, the 
true entropy of the message-generating process. Clearly, 


eH? OO (18) 


The ordered average thus gives a lower bound to the 
entropy H’, which may be greater than the true entropy 
of the process H. The equality signs in (18) are all neces- 
sary; In a message-generating process in which each term 
is selected independently from a constant distribution, 
all four entropies are the same. We will next discuss three 
cases in which H’ = H”. In one of these H’ = H as well, 
and predictive coding may therefore be ideal. 


Case I 


Let the message term m; be drawn from a distribution 
which has always the same form, but with a mean which 
is determined by the past message values: i.e., 


Mines Men oo: Mp) 


= Mme pln ee (19) 


In this case, it is clear that averaging with coincident 
means leads to an ordered average, indeed an average error 
term distribution of the same form as the original message 
term distribution. We have the following properties: 

1. The best IT predictor is defined uniquely (except for 
an additive constant) and is p(m;_, --- m,_; ---). It 
yields an ordered average: in the expression (18), H’’ = H’. 

2. The best IT predictor is the same as the best rms 
predictor, and indeed is the same as the best absolute nth 
moment predictor. 
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3. Predictive coding using the best IT predictor can be 


ideal: in the expression (18), H’ = H. 


This case sounds quite special. Actually, however, it 
includes all of the message-generating processes for which 
Wiener’s filters and predictors were specifically designed— 
i.e., the responses of linear resonators to Brownian 
motions—and many more processes as well, in which the 
predictor is not linear, or in which the distribution of the 
error term is not Gaussian, or both. It also approximates 
a large number of processes of physical interest, in which 


the message varies slowly (relative to the sampling time 


interval) and more or less continuously, and the successive 
error terms are chosen from distributions which are not 
identical, but are similar. As the distributions approach 
identity, properties of the process approach the three listed 
above, hence these are good approximations in such cases. 


| Case II 


Let the message term m; be drawn from a distribution 
which is symmetric and unimodal,” but is otherwise an 
arbitrary function of the past history of the message: 


M(p + x: mj) +++ Me; + °°) 
= M(p — a: mt +++ mi; ++) 
M(p + a: mi. +++ mz; -*°) 
> Mo tym, om, ) (20) 
for || <|y |, with 
Pp = pl. ++ ms; ---). 


Here it is again evident that averaging with coincident 
means will give an ordered average. We still have proper- 
ties (1) and (2) listed under Case I: it is clear that the IT 
and absolute moment predictors agree. However, property 
(3) has been lost. The distributions being averaged are 
not in general identical, and the entropy of the averaged 
error term distribution, by the averaging theorem of Part 
I, is in general greater than the average of the entropies 
of the message term distributions, which is the entropy 
per symbol of the original message-generating process. 

Physically, this is a very broad category of processes. 
It may be expected to include most messages derived by 
the time-sampling of a smoothly varying function gener- 
ated by a process which is essentially symmetrical. The 
Brownian processes are again included. 


Case IIT 


Let the message term be drawn from a distribution 
which is zero to the left (right) of a point p, is discon- 
tinuous at that point, and is monotonic nonincreasing to 
the right (left): 


MAG ite, Oo Ne soe) a3 (0) 
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24 Unimodality is defined for our purposes by (20). It means, not 
merely only one peak that reaches the maximum value, but only 
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for 


Dey. (21) 


The point p may be an arbitrary function of the message 
history. Clearly, averaging the error term distributions 
with the points p coincident will give an ordered average. 
Here we have only the first of the three properties listed 
under Case I: the IT predictor does not agree with the 
absolute moment predictors, and in general ideal coding 
cannot be obtained. 

This category is physically important, since there are 
many physical processes which are of an _ essentially 
positive nature. However, it is not to be expected that 
predictive coding will be very useful for processes of this 
sort. For the point p is not usually a function of the 
message history, but is a fixed origin of some kind, and the 
message as originally generated will have the points p 
already coincident—the best IT predictor will be zero or 
any constant, and the entropy of the average error term 
will be the entropy of the average message distribution: 
in the expression (18), H, = H’ = H”. 

These three cases exhaust the processes of physical 
interest for which a predictor yielding an ordered average 
can be obtained. There are additional cases in which this 
can be done, but they are quite special and not likely to 
correspond to a message which has been derived by time 
sampling of a smoothly varying function of time. Indeed, 
most processes of physical interest as possible messages 
derived from such sampling are included in the above 
three cases or in one other, for which the best IT predictor 
can usually be found only by approximation methods. 


Case IV 


Let the message term m; be chosen from a distribution 
which has the symmetry property that: 


Miia ae ° 
= M(-—m; : 
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That is, reflecting the past history terms across the time 
axis causes the reflection of the distribution about the 
time axis. Here we have none of the properties listed in 
Case I: the four quantities in (3) are all distinct, and the 
best IT predictor cannot be explicitly found. 

Mathematically, the class of processes in this category 
is also quite broad; the Brownian processes are again 
included. The physically interesting new cases included are 
those in which some kind of limiting action is present at 
the source of the message, so that the distribution from 
which a message term is selected is more or less skewed 
towards the time axis. Such a situation arises, for example, 
when a normally distributed message term is passed 
through a symmetric nonlinear device. The transfer 
characteristic of such a device, and its effect on a normal 
message term distribution, is illustrated in Fig. 2 (next 
page). Actually, any electronic system will have some such 
nonlinear limiting—although not necessarily symmetrical— 
for sufficiently large signal amplitudes, but the linear 
range may be so large that the limiting is not evident. 

It should be noted that an average message distribution 
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amples of Part I yield an average message distribution 
with finite power. Another example is a process in which 
the message term m; is chosen from a normal distribution 
with fixed standard deviation and with a mean which is 
m;1/(1 + | m;,_, | ). This process is symmetrically limited 
and is included under Case IV, but it is also included 
under Cases I and II, so that the best IT predictor is 
immediately determinable. 


TRANSFER CHARACTERISTIC 
OF TRANSDUCER 


OUTPUT MESSAGE 
TERM DISTRIBUTION 


INPUT MESSAGE 
TERM DISTRIBUTION 


Fig. 2—The effect of limiting on a probability distribution. 


Although in Case IV the best IT predictor, in general, 
cannot be found explicitly, for practical purposes the 
situation is not bad. For, if the distribution of each message 
term is broad, and the differences for different past 
histories are largely differences in skewing, and not in the 
location of the mean, then the savings to be expected 
using predictive coding, even with the best IT predictor, 
will not be large. If, on the other hand, the difference in 
distributions is largely in the location of the mean, and 
the distributions are narrow, then predictive coding will be 
useful in reducing the channel space required as compared 
to zero-order coding, but almost any reasonable predictor 
will be almost as good as the best IT predictor. This will 
be shown in the next section by an example. 


EXAMPLES OF PREDICTABLE PROCESSES 


Example I 


As a first example, consider the Brownian process given 
in Part I, with the continuous distribution 


1 55 3 
Mims ms4) = — exp [—(m; — am;-_,) /20'] 
ov 2 
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This process is an example for Cases I, II, and IV. It 
therefore has the three properties listed for Case I. As 
discussed in Part I, the entropy H of the process is less 
than the entropy H, of the zero-order approximation to 
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it by an amount log (1/ 4/1 — a’): by property (3) of Case 
I, this full savings may be realized by using predictive 
coding. For a near one the savings may be appreciable. 


Example II 


Consider the discrete conditional distribution 


MG =20 for ke<0 or k>o+ 1 


for 0. < kee sale 


where 7 is the value of the preceding message term m;_, , 
and M,,; is the probability that m; will take the integer 
value k after m;_, has taken the value j. Each message 
term is here chosen from a distribution which gives equal 
probability to all the integers from 0 to the integer one 
greater than the preceding message term. 

From the consistency requirement on the average dis- 
tribution M, , 

M, = 2 MiiM,; (23) 

7=0 
it is possible to find M, : the result is the Poisson dis- 
tribution with \ = 1, 


M, = (1/e)1/k! (24) 


This process is an example of Case III. As indicated 
above, since the origin of the process is independent of the 
previous message terms, a best IT predictor is zero or any 
constant; predictive coding and zero-order coding will 
amount to the same thing, and the channel space required 
will be that required by a source whose entropy per 
symbol is the entropy of M,, . The process also indicates 
the possible ambiguity of the IT criterion for predictors. 
Since the distributions being averaged are all flat, there 
is a rather wide range of predictors which will yield an 
ordered average error term. In particular, the means of 
the distributions (or one of the two integers nearest to the 
mean in the case of even 7) may be aligned, giving the 
best rms predictor, and the resultant average distribution 
will have the same monotonic normal form as does M, , 
and thus the same entropy. The best rms predictor is thus 
a best IT predictor. This will not be true for typical 
examples of Case III, in which the distributions being 
averaged are strictly monotonic: in such processes there 
is no ambiguity. 


Kxample III 


Consider the discrete conditional distribution described 
by 


M,; = b1 + 9/N), k=jg-1 
ib Ppp k=j 
OCLs ifND) ie Ree) tal 
0, k<j—-1 o k>j+1. (25) 


This is a symmetrically bounded process, in which m; 
cannot exceed N. The average distribution M, is again 
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determined by the consistency requirement (23): it turns 
out to be independent of the parameter b: 


M, = (1/2™)2N)'/(N + RUN — 2! (26) 


which is the bionomial distribution of order 2N. 

Using m;_, as a predictor for m; , the average error 
term distribution £, turns out to be a three-term dis- 
tribution: 


Beeb. oat 
pk =O 
b, peer 
0, k<-1l or k>+l. (27) 


The nine conditional distributions for N = 4, and the 
average error distribution, are shown in Fig. 3. Here the 
parameter b has been taken as 1/4. For this value of }, 
the entropy of the average error distribution L, is 1.5 bits. 
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Fig. 3—Message term and error distributions for Example III. 


The entropy of the binomial distribution of order 2N is 
very nearly 4 log,N + 14 bits, and the savings in channel 
space which may be obtained by using predictive coding 
rather than zero-order coding, is given by R, the ratio of 
these two entropies: 
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For smaller values of 6b, the ratio is still smaller. For 
H, , the entropy of the average error distribution, we 
have the approximate expression 


= —2b log, b + 2b(1 — 2b) log, e, bial 


= 2b log, a 
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This gives 


. 4b log, (e/b) 
~ 3+ log, N’ 


This process is an example of Case IV. The predictor 
m; = m,-, does not yield an ordered average. However, 
the difference between the entropy of the average error 
term, using this predictor, and the entropy of an ordered 
average error term, is small for large N. Indeed, for 
b = ¢and N = 1, the entropy of H, is 1.5 and the entropy 
of an ordered average is 1.406, so that the difference is 
already of little practical importance. It may also be 
shown that predictive coding is asymptotically ideal for 
this process for large N; for large N the process becomes 
very much like a process in Case I, in which all terms are 
drawn from identical error distributions, for the prob- 
ability of m;_, being very far from 0 becomes very small, 
and if m,;_, is near 0 then the distribution of m; is nearly 
the symmetric three-term distribution of (27). 

The properties of this process for large N show the 
meaning of the statement made in the discussion of 
Case I, to the effect that distributions which approxi- 
mated the condition (19) also approximated the properties 
(1), (2), and (3). It also illustrates the savings in active 
memory. If 6 is quite small, there will be long sequences of 
successive message terms which are the same value. For 
Shannon-Fano coding to be efficient, it will be necessary 
to take a sequence of message terms so long as to be 
representative of the process, and the active memory 
requirement will thus be very large. However, with 
predictive coding, only one message term need be re- 
membered to give a good IT prediction as to the value of 
the next term. This is perhaps clearer in connection with 
this example, which is already quantized, than it is in 
connection with the continuous process of Example I. 

The examples given have all been Markoff processes of 
the first order. Interesting higher-order processes are easy 
to define, but it is very difficult to determine the average 
message distribution for them, and to evaluate the 
entropy of the process, the entropy of the average message 
distribution, and the entropy of the average error term 
distribution for a reasonable predictor. This sort of 
computation is possible for a Brownian process and for 
virtually no others, except by numerical methods. How- 
ever, this is not too serious a handicap for the application 
of predictive coding, for there are very few message- 
generating processes which are known a priori. The 
significant practical problem is to measure the statistics 
of the process and find the best IT predictor or a reason- 
able approximation to it. This will be difficult if more than 
one or two past message values significantly affect the 
predictor, as discussed in Part I. However, an experimental 
variational approach may be used, in which a trial pre- 
dictor is selected and the average error term which results 
from the use of this predictor is measured. The predictor 
is varied to minimize the entropy of the resultant average 
error distribution. Not only is this a method for finding 
the best IT predictor for predictive coding; it is also a 
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method for finding an upper bound to the entropy of the 
process being measured. Using the relations (18) from 
right to left, we have H’ > H” > H. The entropy of the 
average error distribution of a trial predictor, Hz , is 
larger than the entropy of the best IT predictor, H’, by 
definition, so for any trial predictor, Hz is an upper 
bound for the entropy H of the message-generating 
process. While the upper bound provided may not be 
very good, for such processes as television and facsimile 
it would be very helpful, since there are at present avail- 
able only widely divergent guesses as to the entropy per 
symbol of these processes. 


Coping Mrruops ror INDEPENDENT Mrssace TERMS 

Predictive coding treats successive error terms as if 
they were uncorrelated: i.e., as if the only correlation 
between successive message terms were due to changes 
in the mean, and not in the form, of the message term 
distribution. As we have seen in the preceding examples, 
this procedure is sometimes ideal and often efficient. 
That is, it leads to an average error term distribution 
which, in many cases of interest, may be expected to 
have only slightly more entropy than the true entropy 
per symbol of the original message-generating process. 
This leads us to a consideration of the general problem of 
the efficient coding of a message having uncorrelated 
terms. The discussion will be restricted to quantized 
messages. For practical purposes, we are interested in 
coding methods which either are non-nmemonic or 
required only small codebook memories. It turns out 
that the problem can be solved in one or another of two 
ways, depending on the distribution from which the un- 
correlated message terms are chosen. 
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If H, the entropy of the message term distribution, is 
large compared to one bit, then Shannon-Fano coding of 
each message term will be efficient. For from (14) in 
Part I, we have 


PSS Tee Saal, (28) 
since Gy = HA for a message with uncorrelated terms. 
This gives an efficiency R > H/(1 + H) which is near 
one for H > 1. Since we are coding only one message 
term at a time, the codebook memory has only M entries 
for the M possible quantized values of the message, and 
may conveniently be realized by such devices as the 
binary coding tube’. Eq. (28) implies that for H = 1 it 
is possible to have an efficiency as low as 50 per cent, but 
actually it can be shown that this is not the case, and 
that for H even somewhat smaller than 1, an efficiency 
of 66-2/3 per cent can always be obtained. For most 
applications this efficiency will be high compared to 
other inefficiencies in the system; thus Shannon-Fano 
coding of individual error terms solves problem for H > 1. 


Coding Rare and Random Events—The next category that 
must be considered is message term distributions having 
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an entropy H < 1. For this kind of message, Shannon- 
Fano coding of single message terms will not be useful, 
since each message term cannot be coded with less than 
one output binary digit, giving an H, > 1, which will be 
highly inefficient for small H. Indeed, it is clear that no 
method of coding single message terms into binary digits 
can be efficient. It is necessary to code a large number of 
terms at one time to obtain an output of much less than 
one binary digit per symbol. To avoid a large codebook 
memory, the coding must be non-nmemonic. That this 
is possible is due to the fact that a low-entropy discrete 
distribution must have one high-probability term. Indeed, 
even for H = 3, the largest probability must be > 0.89.” 
Low-entropy messages with uncorrelated message terms 
will then always consist of long runs of a single high- 
probability symbol, interrupted by occasional low- 
probability symbols. We will first consider coding methods 
for the simplest process of this type, in which there are 
only two symbols, and will then show that the general 
low-entropy distribution may be coded efficiently by a 
combination of these methods and Shannon-Fano coding. 

Consider a sequence of zeros and ones. We assume that 
the successive values of the sequence are statistically 
independent. Let the probability of a zero be p, and the 
probability of a one, 1 — p = e. We will define a run of k 
zeros as a segment of 100---00 of the sequence consisting 
of a one, which indicates the start of the run, followed by 
k zeros and a second one, which terminates the run. The 
second one is not counted as a part of the run, since it will 
be counted as the first symbol of the next run. Note that 
by this definition a succession of 7 ones is counted as 
J — 1 successive runs of zeros, each of length zero, followed 
by a run of zeros of some greater length. 

To find the distribution of runs of zeros, we note that, 
given that a segment of the sequence 7s a run, we know 
that it starts with a one, and the probability that it has 
just k zeros is the probability that it has k zeros followed 
by a one. The distribution function for runs of zeros is 
thus 


Se ye 


We may check that this is a reasonable probability 
distribution: 


(29) 


foo) 


De Sh) =F pe ep al 
k=0 k=0 
and find the mean number of zeros per run: 


= ES) = (1p) Dk = vl —p). G0) 


To code the sequence we describe each run by giving a 
binary number that indicates the number of zeros in the 


» A two-term distribution with one probability P, = 0.89 has an 
entropy H = 3. Any distribution with more than two terms and with 
the largest probability P. <_0.89 will have more entropy than this 
by averaging Theorem II, Part I, since it can be obtained by an 
averaging operation on the two-term distribution. Therefore, if anv 
distribution is to have H = }, it must have P, > 0.89. ; ; 
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run.’° The code number for a run of k zeros is obtained by 
omitting the first digit of the binary number k + 1.7” 
The code for the sequence 


100000000 1000000 10000000 100000 1000000000 
is thus the sequence 


,001,11,000,10,010. (31) 


The coded version is shorter than the uncoded version, 
and this discrepancy will evidently increase with the 
average run length. And the difference in length would 
be even greater if it were not for the commas in the coded 
sequence. This is not a trivial matter—the commas, 
or some equivalent for them, are necessary, since we must 
know how the sequence of zeros and ones in the coded 
output is split up into code numbers describing individual 
run lengths. Because of the commas, the output is in a 
ternary, not a binary code, and each comma counts as 
one ternary output digit. 

To find the average number of coded output digits for 
a run of zeros, note that it takes one comma for each run, 
no additional digits for a run of length zero, one additional 
digit for a run of length one or two; in general 


Ck) =n+1 (32) 
digits for a run of length k, where 
n-1 n 
2 Se See 
j=0 j=1 
Now 
eee 2 = 1s De ae eee s)) 
j=0 f=1 
Using (29), (82), and (33) we can find C, the average 


number of coded output digits per run, 


C= SS C(k) S(k) 


1+ (0 - poll op +1 + St: 


| +n > | 


V- (4 — p)/pll0/ — p) 4p /A —'p) + 
+ pl =p) +] 


_ %Shannon (Shannon and Weaver, op. cit., p. 33) discusses the 
problem of coding rare and random events as an example for discrete 
coding. He suggests sending a special code, such as a sequence of 
zeros, for the rare event, and using as a run-length code the number 
of zeros in the run, expressed in a modified binary number system that 
skips all numbers in which the special sequence occurs. He remarks 
that the procedure is asymptotically ideal for p near 1, i.e., as the 
rare event becomes rarer, providing that the length of the special 
sequence is properly adjusted. The modification used here is simpler 
to deal with analytically, and simpler to realize physically in a coding 
circuit. It also has the property of asymptotic ideality, as do a number 
of variants. 

27 The first digit on the left of a binary number is always a one, and 
supplies only positional information. This information is supplied by 
the comma, in the sequence (31). It is necessary to add one to the 
number before removing the first digit in order to provide a repre- 
sentation for a run of length zero. In this code, such a run is indicated 
by two successive commas. 
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Cesta Lay = (34) 


C is the number of ternary output digits per run. The 


equivalent number of binary digits per run is 
C log, 3. (35) 


From (80) the average number of zeros per run is 
p/(1 — p). There is a one at the start of each run, so the 
average number of input binary digits per run is 


We tba IEA i) 
and the entropy per symbol of the input is, by definition, 
p log: Cp) + 1 = p) logs [V/s pe?) 


Thus the input entropy per run is the product of (36) and 
(37); and dividing this by (35) gives the efficiency or 
relative entropy R of the coding process 


R — PAoe U/p) + CL — p) log. [1/1 = p)], 
(1 — p)C logs 3 


(36) 


(38) 


This coding process is the second of a family of similar 
processes. The mth such process uses m symbols to code 
the lengths of runs of zeros in an m-ary code, the (m + 1)st 
symbol being reserved for the commas in the coded message 
(31). For m = 1, the coding process reproduces the input 
signal. For m = 3, the coded output may be pairs of 
binary symbols, with the pair 00 being used for the 
commas. Larger values of m yield codes that are more 
efficient for very small « = 1 — p. Using the summation 
formulas 


> m = (m" — 1/(m — 0); 


0 
n 

pen Se 
Lm = 
g=1 


in place of (32), a derivation like (33) gives the relations 


(m"** — 1)/(m — 1) - 1 


C(m, p) = 1+ (1/d Dg”, with q= pV" — (89) 
(1 — p)C(m, p) log, (m + 1) 
of which (34) and (88) are the special cases m = 2. It is 
shown in the Appendix that 
lim R(m, p) = log m/log (m + 1). (41) 
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Therefore an m may be chosen to give as high an asymp- 
totic efficiency as may be desired. As in the case of m = 3, 
some of the higher m values may conveniently be obtained 
by using combinations of binary or ternary digits to make 
up the necessary m + 1 message symbols: m = 7 is the 
next highest value realizable directly in binary digits.” 


% m-ary digits with m ~ 2” — 1 may, of course, be recoded into 
binary digits. This takes an amount of codebook memory and gives 
an efficiency of translation, which are as usual inversely related. 
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In Fig. 4, R(m, p) is plotted against the logarithm 
to the base two of the average run length, log, [1/(1 — p)], 
for m values of 1, 2, and 3. Each curve has a broad maxi- 
mum, which moves to the right as m increases, the 
maximum being located approximately at the point where 
C(m, p) = m + 1. This is to be expected, since at this 
point the frequency of the special symbol that indicates 
the start of a run will be equal to the average frequency 
of the other symbols. From the maximum, each curve 
approaches its asymptotic efficiency from above. The 
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Fig. 4—Efficiency of non-nmemonic coding. 


expression (41) for the asymptotic efficiency is the obvious 
statement that for p very near 1, the (m + 1)st symbol 
which indicates the start of a run contributes very little 
information, since the runs are long and the symbol 
occurs very seldom. Note that for any value of p, one of 
the three codes shown will have an efficiency R > 79 
per cent. 
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Returning to the general low-entropy distribution, let 
the message terms be so labelled that M, is the term with 
high probability, and the rare terms are M,,1<k <n. 
Then define 


7=1 


P, = M,/e. 


The message may then be divided into two messages: one 
is a two-state process, with probabilities p = 1 — e€ and 
e, and the other is an n-state process with probability 
distribution P, . The two-state process may be coded by 
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the method described above, using m symbols to code 
lengths of runs of the common event and an (m + 1)st 
symbol to indicate that a rare event has occurred, ending 
one run and starting the next. “The” rare event here is 
actually a signal that one of the low-probability symbols 
has occurred, and a term of a Shannon-Fano code for the 
probability distribution P, follows the special symbol, 
to indicate which term of this distribution ended the run. 
Note that the Shannon-Fano coding need not be in the 
m-ary system used for the run-length codes. If an m 
value of 1 or 3 (or any other m = 2’ — 1) is used, and the 
m + 1 symbols used are actually combinations of binary 
digits, then the Shannon-Fano code may be in ordinary 
binary notation. Since in Shannon-Fano coding the end 
of each coded term can be located by inspection, there 
will be no possibility of confusion. 

The entropy of any low-entropy process is divisible 
into two terms that represent the two processes given 
above. We have 


Jel = —> M; log M; 
7=0 


—M, log M, — >) (eP,) log (Pi) 


—M, log M, — e loge —-e Ps log P,; 
1=1 


—(1 — & log (1 — «) — € loge — Sy log P; | 


I 


A, =e eH, ) (42) 


where H, is the entropy of the two symbol process and 
H,, is the entropy of the distribution P, . From (28) we 
have for H’ , the average number of binary digits per 
symbol in the Shannon-Fano code of the n-state process, 


Dae (43) 


For the two-state process, we have by the definition of R 
in Part II, 


Using these two bounds and (42), the over-all efficiency 
of the combined coding is thus 


ae H, + eH, H, -- elf, 
H,-- cH, = Ho Rp) cle 


R’ (44) 
and finally, since R(m, p) < 1 and all the terms are 
positive, 

H 


ig! “ : 
a H./R(m, Pp) =e € 


(45) 


Using only m = 1 and m = 3, taking the higher efficiency 
of the two,”” and noting that H, is determined by «, 


H, = —e loge — (1 — ©) log, (1 — 6), (46) 
22 We are restricted to the use of a coding scheme that gives binar 
digits directly in evaluating (44) and (45), for the expression (43) ie 
valid only for coding into, binary digits. In the general case, coding 
into m-ary digits, the equivalent relation is H,’ < H, + logs m bits. 
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we have that R’ =. 66.7 per cent at « = 4. For smaller e, 
R’ never goes below about 65 per cent, and as « > 0 
(p — 1), R’ approaches the asymptotic efficiency of the 
two-state coding scheme. This may be seen by noting 
that in the limit of small e, (46) becomes” 


Ho =. log, (é/¢). (47) 
Substituting this in (45), 
R’ Ss log. (e/e)R(m, Pp) 
~ log, (e/e) + R(m, p) 
and 
lim R’ = lim R(m, p) = log m/log (m + 1), (48) 
e>0 «0 


where the right side of (48) is given by (41). This is an 
asymptotic efficiency of 79 per cent for m = 3, and can, 
of course, be made as high as is desired by choice of 
sufficiently large m. For « = 4, the entropy per symbol 
of the process H, > 1, by an argument like that in foot- 
note 5, so we have a coding method with an efficiency 
R’ > 65 per cent for all messages with successive message 
terms independently selected and with entropy H < 1. 


APPENDIX 


Asymptotic Efficiency of Run-Length Coding 


Expressions for C(m, p) and R(m, p) for p near one 
may be found by means of the exponential integral 


OV = [ eae, 


First, for qg less than one, g” is a monotonic decreasing 
function of x, so we have the upper and lower bounds 


ve gq” dz.> aes ail (rence = le. 
0 n=1 iL 


Making the substitution z = 
logarithms, gives 


U = (1/log m)EI [log (1/q)] 
L = (1/log m)EI[m log (1/q)]. 


(49) 


m’log(1/q), using natural 


(50) 
Now from (39), for p near one, 
C= Pp mio meee seep) syle 
Using this in (50) gives: 

U = (1/log m)EI((1 — p)/(m — I] 

L = (1/log m)EI[m(1 — p)/(m — 1]. 


Using the series expansion of HI and keeping the first 
two terms,”” 


1/(m-—1) ae (1 


% Franklin, ‘Treatise on Advanced Calculus,” John Wiley & 
Sons, Inc., New York, N. Y., p. 572; 1940. 
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Tf yh log (ms log ee 
log m 
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where y is Euler’s constant, y = 0.5772--- . 
From (39) and (50) we thus have, for p near one, 


U+1>C(m,p) >L+1=U. (51) 
From (36) and (37) we have the input entropy per run, 
converting to natural logarithms: 


VU AG ated) iat (Ces) Mem fl GO PALAC = Ja) 
+ p log (1/p)]. 


For p near one, this becomes 


H/(—"p) "= lossless op) 


Recalling that each output digit is (m + 1)-ary, and 
is equivalent to log(m + 1) natural units, we have, for 
the efficiency R(m, p) for p near one, 


log [e/(1 — p)] log [e/A — p)]_ 
U log (m + 1) (U + 1) log (m + 1) 


Taking the limit gives the same result for each of the 
bounds so 


lim R(m, p) = log m/log (m + 1) 
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Dp) = 


which is (41), the relation to be proved. 
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The Linear, Input-Controlled, Variable-Pass Network’ 


B. E. KEISER{ 


Summary—tThis paper describes the study and development of a 
linear, variable-pass network system which is controlled by the 
Fano short-time autocorrelation function of the input. Given an 
input function, the message, whose short-time power spectrum 
varies in an unpredictable manner with time, and to which there has 
been added a different function, the disturbance, whose short-time 
power spectrum is either time-invariant or varies in a completely 
known manner, a linear, input-controlled, variable-pass network 
can be specified which minimizes the mean-square error between 
the message input and the total output, taking network delay into 
account. Methods for mathematical computation of the mean-square 
error have been devised. 

The linear, input-controlled, variable-pass network has been 
found to have a lower mean-square error than that attainable with 
an optimum-mean-square, linear, fixed, selective network, for cer- 
tain types of input messages. 


INTRODUCTION 


ECENT years have witnessed the appearance of 
R an extensive amount of literature on the theory 
of variable-pass networks.’ Very little research 
has been done, however, on the capabilities of such net- 
works in improving the transmission of information. This 
paper shows how a variable-pass network may be superior 
to a fixed, selective network in the reduction of the dis- 
turbance which often accompanies a desired message. 

Let the desired message, expressed as a function of time, 
be denoted by /;,(¢). Suppose that a disturbance, f(t), 
has been added unavoidably to the message. Let the sum, 
fi + fied, be denoted by f,(t). The function, f;,(¢), is 
regarded as the input to a selective network whose purpose 
is to operate upon f;(¢) in such a manner that the network 
output, call it fo(@, is as nearly like the desired message, 
f(t), as is possible. 

In terms of the frequency domain, a reduction in the 
bandwidth of the selective network generally reduces the 
amount of disturbance transmitted by it.* Too severe a 
reduction in bandwidth, however, generally produces 


intolerable distortion of the message. Therefore, one must’ 


decide what type of performance characterizes an optimum 
system for the problem at hand. 


* The material presented in this paper is based upon a dissertation, 
“Tmprovements in Information Transmission Attainable With a 
Linear, Input-Controlled, Variable-Pass Network,” submitted in 
partial fulfillment of the requirements for the degree of Doctor of 
Science in Electrical Engineering at Washington University, St. 
Louis, Mo., January, 1953. This research was supported by an 
R. C. A. Fellowship in Electronics, administered by the National 
Research Council. The research reported here was the subject of a 
seminar presented in the Electrical Engineering Department of 
Washington University on January 23, 1953. 

+ White-Rodgers Electric Co., Saint Louis 6, Mo. 

1 J. R. Carson and T. C. Fry, ‘‘Variable frequency electric circuit 
theory with application to the theory of frequency modulation,” 
Bell Sys. Tech. Jour., vol. 16, pp. 518-540; October, 1937. 

2. A. Zadeh, ‘‘Frequency analysis of variable networks,” Proc. 
IRE, vol. 38, pp. 291-299; March, 1950. 

3 J. R. Carson, “Selective circuits and static interference,” Trans. 
ATEE, vol. 43, pp. 789-796; 1924. 


One criterion that has been discussed extensively in the 
literature,* and is relatively easy to deal with mathemati- 
cally, is the minimization of the mean-square error, L,, . 
By definition, 


[fo(t) ea fa) dt (1) 


1 he 
a TH0 27 = l! 
is the mean-square error of f;,(¢) and f,(é). If “a” denotes 
the delay encountered by f,,(¢) in being transmitted 
through a selective network, an alteration in (1) is 
required in order that erroneous results not be obtained. 
The mean-square error of f;,(¢ — a) and fo(¢) is 


9 


Tox + 


Be = lim = _. [fo(t) = fialt ory. a)| dt. (2) 


Practically speaking, the limit in (1) and (2) is taken as a 
value of 7 sufficiently large that essentially no change in 
the results would be obtained by taking a larger value. 

The principal objective of optimum-mean-square, 
selective network theory is the minimization of expression 
(2). Ordinarily the apparent power density spectrum, 
®(w), or the autocorrelation function, ¢(7), of the input 
message and disturbance are used to specify a time- 
invariant system. However, optimum-mean-square, selec- 
tive network theory may be extended to the case of 
variable-pass devices, as will be shown in this paper. 

The purposes of the investigation reported in this 
paper were: 

1. A specification of the type of input functions which 
are capable of being processed more suitably by a variable- 
pass device than by a fixed, selective device. 

2. The determination of the response of the variable- 
pass device which minimizes the mean-square error 
between message input and total output, provided the 
device is controlled by the Fano short-time autocorrelation 
function of the input. 

3. A comparison of the mean-square error performance 
of an optimum, variable-pass device with that of an 
optimum, fixed, selective device designed to process the 
same input functions. 


FUNCTIONS wiTH TIME-VARYING SPECTRA 


The conventional definitions of the apparent power 
density spectrum, &(w), and the autocorrelation function, 
o(r), require that average values involving the time 
function, f(¢), be taken over all values of time ¢. If f(é) 
is not periodic, but if 


, 1 = 
Neen / 2 
pe a oer BON as ee) 

*N. Wiener, “The Extrapolation, Interpolation, and Smoothing 


of Stationary Time Series,”” The Technology Press, John Wil 
Sons, Inc., New York, N. Y., 1949. oy, » John Wiley and 
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‘then, by definition,> 


wo) = Lim df soe a at (3) 
) and 
(7) = lime [fost — 9 a (a 


The nature of f(¢) over an interval of t small compared 
with the entire interval is ignored completely when 
such definitions are used. A given f(t) may possess 
markedly different properties over different ranges of 
time ¢ as, for example, is evidenced by the fact that 
spectrograms presenting the frequency spectrum as a 
function of time have become very useful tools in the 
study of speech waves. A group of investigators’ at the 
Bell Telephone Laboratories working on “visible speech” 
has found that such spectrograms can be read by trained 
people and therefore must contain much pertinent lin- 
guistic information regarding the speech wave from 
which they are obtained. 

In order for a spectrum or correlation function to vary 
with time, more weight must be given to values of the 
function occuring at some times than at others. Although 
there are many ways in which short-time correlation 
functions and power spectra might be defined, Fano’s 
definitions’ appear to be the only ones that can be handled 
with ease mathematically and experimentally. The 
short-time autocorrelation function, ¢,(7, @), is defined as 


b(r,@) = 20 f f@fe— Aer? de. ©) 


The following discussion characterizes the type of 
input functions for which, in some cases, a variable-pass 
“network proves superior to a fixed, selective network. 
Let a function with a time-varying frequency spectrum 

denote one whose short-time power spectrum is more 
sharply defined than is its long-time power spectrum at 
some or all times. The words “more sharply defined” 
should be given the following physical interpretation. 
Given two time functions whose frequency spectra have 
been determined on either a short- or a long-time basis, 
let the areas under each spectrum curve be made equal 
by a normalization process. The spectrum which then has 
the greater maximum ordinate will be referred to as being 
“more sharply defined.” In general, the more sharply 
defined the message spectrum is, the lower the mean- 
square error can be made. 

If a spectrum is more sharply defined on a short-time 
basis than on a long-time basis, a selective network 
operating on a short-time basis and controlled continuously 
by the present and immediately past statistical properties 
of the input is capable of separating a desired message 

5Y. W. Lee, “Application of Statistical Methods to Communica- 
tion Problems,” Technical Report No. 181, Research Laboratory of 
Electronics, Massachusetts Institute of Technology, Cambridge, 
Mass., pp. 8- 9; September 1, 1950. 

'R. R. Riesz and L. Schott, “Visible speech cathode-ray trans- 
lator,” Jour. Acous. Soc. Amer., vol. 18, pp. 50-61; January, 1946. 

18 M. Fano, “‘Short-time autocorrelation functions and power 


spectra,” Jour. Acous. Soc. Amer., vol. 22, pp. 546-550; September, 
1950. 
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from a disturbance more completely (on a mean-square 
error minimization basis) than would be possible with a 
fixed network. This is true since the fixed network neces- 
sarily must operate on a long-time basis. This requires, 
of course, that the message or disturbance, or both, be 
functions with time-varying spectra. The rate at which 
the spectral properties of the input .vary, governs the 
choice of the fixed weighting parameter a. 

In general, functions with time-varying spectra have 
two types of variations, one of which characterizes rapid 
fluctuations of the functions, the other, slow spectral 
changes. A radio frequency carrier, phase or frequency 
modulated at a variable audio frequency rate, is an 
example of a function with one type of time-varying 
spectrum, that of time-varying frequency. Another type 
of function with a time-varying spectrum is encountered 
in the case of one with time-varying bandwidth. The fact 
that much phonograph music is of this type led Scott. to 
design a variable-bandwidth amplifier,” bandwidth being 
controlled by reactance tubes which act in accordance 
with the presence or absence of certain frequency com- 
ponents in the input. This device, known as a dynamic 
noise suppressor, really is an input-controlled, variable- 
pass network, although it was not designed for the purpose 
of minimizing the mean-square error or. for making 
optimum use of spectral properties of the input through 
correlation techniques. 

The purpose of using Fano’s short-time correlation 
function for controlling the variable-pass network is to 
detect slow spectral variations of the input message, 
fis(Q), which is assumed to be a function with a time- 
varying spectrum. Once the slow spectral variations have 
been detected, the transmission characteristics of the 
variable-pass network can be controlled accordingly. 


INPUT 
TRANSLATOR 


OBSERVER 
(CONTROL NETWORK) 


b,j (t,«) 


CONTROLLED 
VARIABLE-PASS 
NETWORK 
H(jw;t) ee 


fo (t, ) 


Fig. 1—Essential features of an input-controlled, variable-pass 
network. 


ESSENTIAL FEATURES OF AN INPUT-CONTROLLED, 
VARIABLE-Pass NETWORK 


Fig. 1 is a block diagram showing the essential features 
of an input-controlled, variable-pass network. Its three 
chief components are shown there. 

8H. H. Scott, ‘“The reduction of background HOE in the repro- 


duction of music from records,” Proc. N.E.C., vol. 2, pp. 586-596; 
1946. 
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The “input observer’ is a Fano short-time auto- 
correlator; it obtains ¢,;(7, @), the short-time auto- 
correlation function of /,(¢). The “translator,” or control 
network, is the device which translates ¢,;(7, @) into 
control functions for the variable-pass network. In practice, 
the “translator”? may consist of several time-invariant 
amplifiers with specified nonlinear amplitude character- 
istics. The reason for the nonlinearity is that, in general, 
there is no guarantee that the control functions for the 
variable-pass network itself will be linear functions of 
dix (7, @). 

The transmission function of the variable-pass network 
is denoted by H(jw;¢). The required manner of variation 
of H(jw; t) with ¢,;(7, a) for mean-square error minimiza- 
tion is the subject of a later section of this paper. 


NEcESSARY RESTRICTIONS ON THE INPUT MESSAGE 
AND Input DistuRBANCE FUNCTIONS 


Now that a general picture of the operation of an input- 
controlled, variable-pass network has been obtained, the 
next step in its theoretical development is a specification 
of the restrictions which must be placed on the input 
message and the input disturbance in order to obtain 
time-varying control functions for the variable network. 
Although the input message is referred to as f,,(¢) and 
the input disturbance as f;.(¢), the choice of subscripts 
1 and 2 is entirely arbitrary, and in the material which 
follows, the roles of the two, as well as the restrictions 
placed upon them, may be interchanged entirely if 
desired. In other words, the variable-pass network may 
be controlled on the basis of time variations in the short- 
time autocorrelation function of the disturbance rather 
than that of the message. 

Let the assumption be made that /;,(¢) is a function 
whose spectrum is either time-invariant, or one whose 
spectral variations are completely known for all values 
of time ¢. Furthermore, f;,(¢) and f;.(¢) are assumed 
to be uncorrelated with one another on a long-time basis, 
and essentially uncorrelated also on a short-time basis 
(specified by a). Then 


dii(T, a) = dria(T, a) cf dri2lT, a). (6) 


Eq. (6) permits ¢,;:(7, @) to be ascertained from an 
observation of ¢,;(7, a). 

An example of a case where ¢, ;2(7, a) 1s essentially time- 
invariant is obtained by letting f;.(¢) be very wide-band 
random noise having no detectable spectral variations. 
For an example of a ¢;;2(7, a) which corresponds to an 
fi2(t) whose spectral variations are ‘completely known,” 
consider the following case. Suppose that in the frequency 
domain the function f;,(¢) occupies region A only, but 
that function f;.(¢) occupies both region A and region B, 
where A and B are mutually exclusive regions. If the 
spectral variations of f;.(¢) are such that they can be 
determined completely by an observation that is limited 
to region B, then the results of this observation in region 
B can be utilized in the autocorrelation and message 
separation process being carried out in region A. In 
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terms of (6), since ¢,;(7, @) is observed directly in region 
A at each value of time ¢, and since ¢,;2(7, a) is assumed 
to be known for all values of ¢ from its behavior in region 
B, the present value of ¢,;:(7, «) can be obtained at each 
instant of time. 

The requirement that f;,(f) and f;.(t) be “essentially 
uncorrelated”? on a short-time basis specified by a can 
be shown to be satisfied for all practical purposes provided 
f(t) and f;2(¢) are not sine waves separated by less than 
a/m cycles per second, or provided they are not random 
functions whose spectra are concentrated’ in a common 
bandwidth of the order of magnitude of a/z. Zero long- 
time cross-correlation, of course, also is required. 


A THEOREM RELATING THE AVERAGE VALUE OF A 
WEIGHTED FUNCTION TO THE AVERAGE OF THE 
FUNCTION 


In comparing an input-controlled, variable-pass net- 
work with a fixed, selective network, one generally is 
interested in the long-time behavior of each. A theorem 
of considerable use in relating short-time properties of 
functions to their long-time properties is the following: 

Theorem I-A: if F(t) is a bounded, continuous, single- 
valued function of ¢, and if its derivative, F’(t), is con- 
tinuous, then 


; | ‘ Ce —2a(t—z) 
lim sa F(t) dt = Lia I F(a)e dx dt 


Paton) xa 
0<a< ~,areal. (7) 


This theorem may be proved by use of integration by 
parts and by application of the mean-value theorem for 
integrals. The theorem may be used to obtain the re- 
lationship between the short-time cross-correlation of 
two random (but bounded) functions. (If these functions 
are voltages or currents, they are said to be of the finite 
power type.) 

Let F(t) = fa(t) fea(t — 7). Application of (7) yields 


1 T 
b4,n(7) = Lim oT he $:a,B(7, a) at. (8) 


If fs() = fae(t), then the expression obtained is a re- 
lationship between short- and long-time autocorrelation 
functions, and the subscripts A and B may be dropped. 
Thus 


Ren (By \ 
(7) = lim 5m [ di(r, a) al. ©) 
If F(é) is such that 
: 1 2 
Pp aL F(t) dt = 0, (10) 


Theorem I-A still is valid, but is of little direct use. How- 
ever, a relationship similar to Theorem I-A may be 
proved. 

Theorem I-B: if F(t) is a single-valued, bounded func- 
tion for which 


| F(t) dt <0} 


955 
shen 
[ riwat=20f fret? dea 
OR an amoe cared ls (11) 
Letting F(t) = f.(@) fa — 0) yields 

beac) =f duaalr,a) a (12) 
where, by definition, 

bol) =f f@flt— Dat (08) 
s the long-time cross-correlation of two functions having 


inite energy. If f4(t) = f2(t), then the expression obtained 
is a relationship between short- and long-time auto- 
correlation functions. Thus 

o(r) = | dir,a) at. (14) 


Mean-SquarReE Error IN TERMS OF CORRELATION — 
FUNCTIONS OF THE INPUT AND OvuTpuUT 


The mean-square error of a variable-pass network 
depends not only upon the transmission function, H (jw; t), 
as in fixed, selective network theory, but also upon the 
way in which H(jw; t) varies with time, and thus upon 
b(t, a). Let E, (a) denote the mean-square error of a 
variable-pass network, provided the transmission function 
is the optimum one for a given network delay, ‘‘a,” and 
a given input, and provided this optimum transmission 
function is controlled in accordance with the short-time 
autocorrelation function of the input. Once EF" (a) has 
been found, there still remains the problem of finding that 
value of a for which E£,”’(a) is a minimum. Since the 
output of an input-controlled, variable-pass network 
depends upon a, long-time cross-correlation and auto- 
correlation functions involving the output also can be 
expected to depend upon a. Expansion of (2), with fo(¢) 
replaced by f(t, ) and EF,” replaced by E,,’ (a), applica- 
tion of the definition of the short-time correlation func- 
tion, and utilization of Theorem I-A, lead, under the 
restrictions developed previously, to a proof that 


EN. (a) = $01(0, a) om 201, 11(4, 9) = $02(0, a) a8 $;:(0). (15) 


Eq. (15) was obtained on the assumption that f;,:(é), 
and hence f,(t, a), do not tend to zero as t ~+o. If 
either fi(¢) or f;2(t) or both are functions which do tend 
to zero ast ~+~, then instead of studying the mean- 
square error, the quantity which must be studied is 


[i Wott, 0) — fale = af at. 


Wherever the mean-square error has been obtained, the 
juantity 


[Wott a) = fae = oy at 
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also can be found, provided the given functions satisfy 
Theorem I-B instead of Theorem I-A. (An example of a 
function of this type is a finite train of pulses.) A proof, 
similar to the one for (15), has been obtained that 


[thot ) = fale = ay at 


= $.01(0, &) — 201, 11(4, w) + $.02(0, a) + ¢$.,1(0). (16) 


Eqs. (15) and (16) are similar to expressions which 
have been obtained in the literature for the mean-square 
error of fixed, selective networks. For fixed networks, 
however, no parameter a is involved. Eq. (15) has been 
verified experimentally. 

By means of a theorem relating the short-time cross- 
correlation of two functions to their short-time auto- 
correlations, and by means of a theorem relating the 
short-time autocorrelation of the output of a variable- 
pass network to the short-time power spectrum of the 
input, it is possible to find the mean-square error of an 
input-controlled, variable-pass network in terms of short- 
time autocorrelation functions of the input only. The 
relationships obtained, however, are rather lengthy, and 
therefore are not presented here. 


THE CONTROL OF THE TRANSMISSION FUNCTION, 
(jw; t) 

If the transmission function, H(jw; t), could be con- 
trolled perfectly by values assumed by ¢,;;:(7, @) and 
¢,:2(T, @), the mean-square error of a variable-pass net- 
work would consist only of a message error and a dis- 
turbance error. However, when message and disturbance 
are present simultaneously at the input of an input- 
controlled, variable-pass network, there arises an ad- 
ditional error, which will be called ‘tracking error.” 
When such an error is of a negligible order of magnitude, 
the following results, stated in Theorem II, can be 
obtained. 

Theorem IT: let ®{3;(w, a) be the short-time power 
spectrum of a function with a time-varying spectrum, and 
let ®/33(w, a) be a short-time power spectrum which 
either is time-invariant, or whose time variations can be 
determined by a process independent of the presence of 
fi). If two functions, f;,(¢) and f;.(¢), possess essentially 
zero cross-correlation for all 7 and all ¢, let these functions 
be the input message and input disturbance, respectively, 
to a stable, variable-pass network with a transmission 
function, H(jw; t). Let H(jw; ¢) at each instant of time ¢ 
be given the values, H,(jw; ¢), that an optimum, fixed, 
transmission function would be given for an input with 
conventional power spectrum functions, ®;/;(#, @) and 
@'3(w, a) at t. Then EX (a), which is the sum of the 
message error and the disturbance error of the variable- 
pass network, assumes its minimum value for a given 
“a and a. Instead of using ®)7i(w, a) and ;%3(, a), 
control on the basis of correlation functions may be used 
if, in place of the long-time correlation functions the 


sy: - (a) —alr| (a) 
quantities, e °'"! $/i1(7, a) and e “'"' $,42(7, a), are used. 
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ReSULTS OBTAINED EXPERIMENTALLY 


Fig. 2 shows the basic elements of the experimental 
equipment. The message, f;,(/), consisted of a random 
voltage of bandwidth 60 cycles per second and of center 
frequency between 2.5 and 3.0 kilocycles per second. The 
center frequency was a random function of time. Its 
values fluctuated in an unpredictable manner about 2.75 
kilocycles per second. The disturbance was very wide- 
band random noise possessing no short-time spectral 
variations. 


MESSAGE 


GENERATOR 


F 5, (+) 


DISTURBANCE INPUT- 
CONTROLLED 
VARIABLE- 


PASS NETWORK 


MEAN-SQUARE 
ERROR 
MEASURING 
DEVICE 


GENERATOR 


f;2(t) 


Fig. 2—Basic elements of experimental equipment. 


For the particular combination of message and dis- 
turbance chosen, the variable-pass network showed 
improvements as listed in the following table: 


TABLE I 
RepuctTion OF [fo(t) — fa(t — a)] OBTAINED EXPERIMENTALLY 
Noise-to-Signal | No Network Fixed Variable-Pass 
Power Ratio Inserted Network Network 
Lis 0 db. 271 db. ORord bs 
1.0 0 db. 2esidbs 6.0 db. 
0.75 0 db. 2.6 db. 6.9 db. 
0.50 0 db. 3.1 db. 8.1 db. 
0.25 0 db. 3.5 db. 9.6 db. 


The noise-to-signal power ratio was that measured in a 
500 cycles per second bandwidth. For each noise-to-signal 
power ratio, the optimum-mean-square selective network 
possesses a different bandwidth. In the experimental 
example chosen, the variable-pass network is able to 
operate with a fixed bandwidth of about 100 cycles per 
second, while the corresponding fixed network would 
require a bandwidth of about 500 cycles per second. A 
5-to-1 reduction in bandwidth should yield a decrease in 
noise power transmission of about 7.0 decibels. Table [I 
shows that this value was approached at relatively low 
disturbance levels. The poorer improvements obtained at 


higher disturbance levels may be attributed to tracking | 


error, arising chiefly from the fact that the short-time 


autocorrelator transmits a certain amount of noise. (The. 


noise power transmitted by a Fano short-time auto- 
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correlator is proportional roughly to @ and also to the 
noise power level at its input.) 

Although the example treated experimentally was one 
in which bandwidth was fixed (only center frequency was 
variable), much more complicated types of variation 
could be handled if necessary. The simplifications em- 
ployed permitted a device to be constructed in which only 
two 7 values of the short-time autocorrelation function 
were used in the control of the variable-pass network. 
More complicated spectral variations also might be 
handled by means of a short-time autocorrelation function 
other than Fano’s. However, the mathematical theorems 
developed in this research are valid only when the Fano 
autocorrelation function of the network input is used as 
a basis of control. 


SPECIFICATION OF THE PARAMETER @ 


One question which may have arisen by this time is how 
short a short-time autocorrelation function should be used. 
No precise analytical answer can be obtained in the 
general case without a considerable amount of data. 
However, a simple answer in physical terms can be given 
which should be of value in many cases. Let &(¢) denote 
the varying property of the input message spectrum. 
Thus é(¢) may denote bandwidth, center frequency, or 
some special combination of these. Let a Fourier analysis 
of £(t) be made, thus yielding the spectrum of the variation. 
Choose a such that the Fano correlator’s low-pass net- 
work (bandwidth: a/m cycles per second) passes essentially 
all frequency components of the spectrum of the variation 
without undue attenuation. Aside from this requirement 
the bandwidth of the low-pass network should be kept 
as small as possible. 


CoNCLUSIONS 


A sufficient condition that an input-controlled, variable- 
pass network be superior to a fixed, selective network is 
that the short-time spectrum of at least one of the two 
input functions, f;,(¢) and f;.(¢), be more sharply defined 
than its corresponding long-time spectrum, over some, 
but not all values of time. (By assumption, f;,(¢) and 
fi2(t) possess essentially zero short-time cross-correlation 
at the value of a utilized.) A necessary condition that a 
variable-pass network be superior to a fixed, selective 
network is that at least one of its two input functions, 
fia(d) and f;2(t), be a function with a time-varying spec- 
trum over some, but not all, values of time. 

The extent of the improvement attainable depends upon 
the spectral properties of the input message and dis- 
turbance, 1.e., bandwidths on short- and long-time bases, 
as well as rates of variation. If neither of the two input 
functions have a time-varying spectrum, the variable-pass 
network degenerates into a fixed, selective network. 


MATHEMATICAL Notation Usep 


E,, mean-square error. 
(a) . 
JOSS mean-square error of a fixed, selective network 
having a delay of ‘‘a’”’ seconds. 


(1955 


E,. (a) 


H (ju; t) 


a 

cA) 
f (2) 
fin 
fi2(d) 


Fold) 
Fol, a) 


fol, a) 
foot, a) 


t 
P(w) 


®, wl (w, a) 
@)51(w, a) 


®, :2(w, a) 
@;3(w, a) 
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mean-square error of a linear, input-controlled, 
variable-pass network having delay ‘a’ and 
correlation weighting parameter a. 
transmission function of a variable-pass net- 
work. 

network delay in seconds. 

a function of time. 

total input to network. 

input message. 

input disturbance. 

total output of network. 

total output of a variable-pass network con- 
trolled by a short-time autocorrelator having 
weighting parameter a. 

message component of total output of variable- 
pass network. 

disturbance component of total output of 
variable-pass network. 

time in seconds. 

long-time (conventional) apparent power den- 
sity spectrum. 

Fano short-time power spectrum of /;,(é). 
Fano short-time power spectrum which f ;:(¢) 
possessed “‘a’”’ seconds ago. 

Fano short-time power spectrum of f ;2(t). 
Fano short-time power spectrum which f ;2(¢) 
possessed ‘‘a’’ seconds ago. 

the parameter which governs the weight given 
to past values of a function in determining its 
short-time correlation function. It is equal, 
numerically, to the half-bandwidth of a (short- 
time) spectrum analyzer in radians per second. 


g 


pa (r) 


ba,2(T) 


o.(7) 


p.a(7) 


ea, p(T) 
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a time-varying property of a short-time 
spectrum. 

the time by which one of the functions is de- 
layed in a correlation process. 

long-time (conventional) autocorrelation of a 
function possessing a nonzero apparent power 
density spectrum. 

long-time (conventional) autocorrelation of 
fa(t), where f4(¢) possesses a nonzero apparent 
power density spectrum. 

long-time (conventional) cross-correlation func- 
tion of f,(¢) and f,(t), functions which possess 
nonzero apparent power density spectra. 
long-time (conventional) autocorrelation of 
function with a finite energy density spectrum. 
long-time (conventional) autocorrelation of a 
function, f4(¢), possessing a finite energy density 
spectrum. 

long-time (conventional) cross-correlation of 
fa(t) and f,(¢), functions which possess finite 
energy density spectra. 
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Spectral Power Density Functions in 
Pulse Time Modulation 


H. KAUFMAN} AND E. H. KING{ 


Summary—Spectral power density functions corresponding to 
various types of pulse shapes, and probability distribution functions 
arising in the study of pulse time modulation problems are com- 
puted. The results are presented in tabular form. The following 
cases are considered: PAM and PPM, for arbitrary pulse shape, 
PDM, for rectangular, Gaussian, and error-function pulse shapes, 
and SEM, for rectangular pulse shape. 


INTRODUCTION 


N analysis of the performance of systems composed of 
linear filters and having pulse train inputs, many 
problems consist of determining the behavior of 

such systems when the pulse train inputs have random 
fluctuations in pulse amplitudes, pulse positions, or pulse 
durations. In the setting up of these problems, it is con- 
venient to have available standard types of idealized 
pulse forms, commonly used probability density functions, 
and spectral power density functions for idealized types 
of pulse time modulation. The chief purpose of this paper 
is to make available, in tabular form, a compilation of 
spectral power density functions for various pulse shapes, 
various probability density distributions, and various 
types of pulse time modulation that the authors have 
found to be useful tools in the analytical evaluation of 
systems’ performances. 

The tables do not exhaust all the possibilities, but are 
intended to cover many of the commonly used types for 
which solutions could be found in closed form. The 
derivation of the spectral power density functions for a 
few cases have appeared in the literature." ° MacFarlane’s 
analysis’ served as basis for the derivation of the additional 
spectral power density functions included. A closely 
related study, using the correlation function approach, is 
given by Kretzmer.” 


{7 Mathematics Department, McGill University, Montreal, Que., 
Canada. 

t Laboratory for Electronics, Inc., Boston, Mass. 

1G. G. MacFarlane, “‘On the energy-spectrum of an almost peri- 
odic succession of pulses,’ Proc. I.R.E., vol. 37, pp. 1139-1143; 
October, 1949. See also discussion of this paper by T. S. George, 
Proc. I.R.E., vol. 38, pp. 1212-1213. 

2. R. Kretzmer, ‘Interference Characteristics of Pulse-Time 
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SuMMARY OF FORMULAS 


The direct and inverse Fourier Transforms as used here 
are given by 
(1a) 


Gls) = [10 exp (jut) at 


1) = 5 [  G) exp (jad) de, (1b) 


where f(t) and G(w) are the time function and the spec- 
trum, respectively, of a typical member of the pulse train 
and are given in Table I. (Tables I-VI follow page 41, 
and are printed consecutively.) 

If g(x) is the probability function of a random variable, 
x, the discrete and continuous spectral power density 
functions for pulse amplitude modulation, PAM, are 


Pee = ReOipecnoe ey Sy eT ee. 
(2a) 
P(e) = 7 |G) ) @— XY, (2b) 


where P,(w) represents the line spectrum of the periodic 
components, P,(w) represents the continuous power 
density per cycle per second, w, = 27/7, mean repetition 
frequency, in radians per second, 


xX = iL. xq(x) dx, (3a) 


fo) 


i x q(x) dx — X’. 
(3b) 


@— XP = [ @— x) dr = 


The discrete and continuous spectral power density 
functions for pulse position modulation, PPM, are 


Pile) = F |G) P| X |? le — may), 
m= 0, +1, +2,--- (4a) 
Plo) = 7| GW) P@— XY; (4b) 
where 
X = [exp (jon)q(a) ac) (5a) 


1955 


(e) 


ek) = ike exp (jx) exp (—jwx) g(a) dx 


St eels aXe Sb) 


The discrete and continuous spectral power density 
functions for pulse duration modulation, PDM, are 


ove =F X74 £2 Fite, tay ee ey ee GE 
1 ——s; 
P.(w) = T Ce Oe (6b) 
where 
ers / Ein Due (7a) 
pee = if Carotene Sb 


The discrete and continuous spectral power density 
functions for single edge modulation, SEM, are 
ig a(w) 


2 oN 
= e | X ie dw — mw,), m = 0, +1, £2, --- (8a) 


Pe) = G@— XP (8b) 


where X is given by (7a), and 


foo} 


@—xX¥-[ \Ge9ha@a-|xP © 

In (3a), and (8b), x is the random variable defining 
pulse amplitudes. The q(x) is defined in Table I, taking 
A = 1. In (5a) and (5b), 2 is the random variable defining 
a pulse position relative to the unmodulated position of 
the preceding pulse, and the q(x) is defined in Table II, 
taking A = T. In (7a) and (7b), x is the random variable 
defining the pulse duration, and q(x) is defined in Table 
II, taking A = +. In (9), x is the random variable defining 
the displacement of either the leading edges or the trailing 
edges of the pulses from the positions occupied by the 
centers of unmodulated pulses. The q(x) is defined in 
Table II, where for leading edge modulation, A = — 7/2, 
and for trailing edge modulation, A = 7/2. 

The power contained in the real frequency band ex- 
tending from w, to w, is given by 


Rio, ,w:) = 52 f Pale) + P.)] de 
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+50 [ Pile) + Poe)] de 


es ! im [Poe eects (10) 


EXAMPLE 


The discrete spectral power density function of a 
rectangular pulse train having a constant pulse amplitude, 
HH, a constant pulse duration, 7, and a repetition period, 
T, is given by 
» sin” (wr/2) 


Morjaer ee 


P(w) = a (E'7) Mey) , 


m= 0, +1, +2, ete. 


The average power in this pulse train is given by 


1 foe} 
R(-@, @) =5- | P,e) de 
*y, (iis) af esis (wr/2) > 
mika ees. dm — mw,) dw 
vas (Er)” & sin’ (mw,7/2) ey 
BS ee: 2d (ijt) ee 


DESCRIPTION OF TABLES 


Table I consists of five idealized pulse forms together 
with their Fourier transforms. Note that the pulse length in 
every case except that of the square pulse is defined as the 
time between half-amplitude points of the time functions. 
This lends a convenient symmetry to the Fourier trans- 
forms, and in no way lessens the generality of the results. 

Table II consists of four probability density functions 
frequently encountered in systems’ analyses. 

Tables III to VI present the spectral power density 
functions for various random time modulations. The 
spectral power density functions consist of two parts. The 
discrete spectral power density function, P,(w), represented 
by the Dirac delta functions at the harmonics of the 
repetition frequency, is a measure of the power distribution 
in the periodic components of the pulse train. Similarly, 
the distribution of power continuously spread over the 
frequency domain is given by the continuous spectral 
power density function, P,(w). All six tables mentioned 
above appear on pages 42 through 46 immediately follow- 
ing this paper. 
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TABLE I—Putsr SHApPes AND TRANSFORMS 
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{= al G(w) exp (Gwt) dw, Gw) = ih f(t) exp (—jwt) dt 
Type Graph f(t) G() 
Ad E, Weds)” sin es 
Rectangular 5 - Er 
t 0, al Sco Ss 
-TR ° Th. Dy 


Triangular _# (t — 7), Ohi ms ei 
: call 
2 
0, |b) >< 
E 
coe +), 
aT 2 
T 1 
= Cae =e ee) 
Trapezoidal 
Gaussian 


Error function 
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TABLE IJ—Prorpasiuitry Density Functions, q(x) 


Mean 
Ge Mean Squared 
Type q(x) Graph of ¢(z) Pigeon 
(eer 
2 
Sinusoidal = 
Normal 2 
Flat a 
‘Double Spike A,\(1 — A) 
A, d(x = a) st 5 : G ae 2 
v2 1 2 


Note: All g(x) are normalized: il Oo) da) = 


TABLE IIJ—Ranpom Puuse Ampritupre Mopunation, (PAM) 


iDicto Discrete Spectral Power Density Function, Continuous cider 2: Density 
at 
Function P plo) : 
4 le o(w) 
. . Qa 2 to 
Sinusoidal Zels 2 = 0 2 
T G(w) | d(w My) Ti | G&) | 
Normal IV hla) Vl ak) a | G@) |’ 
ies ss ap 
Flat PTET Cee eve af | G@) 
a T’ ae Be 
2 A, = AVGL— 2), ; 
Double Spike a [A,(a,; — a) + a2]? | GW) |’5@ — me,) i( Me ta) | G@) 
2 
ieee im = O, sell, SEA, ee 


Note: For the double spike case, G(w)is taken from Table I with H = 1. For the other three cases, q(x) as given in Table IT, 
have mean values of 1. 
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TABLE IV—Ranpom Putse Position Mopunation, (PPM) 


Distribution 
Function 


Discrete Spectral Power Density Function, 


P. p(w) 


Continuous Spectral Power Density 
Function, 


P(w) 


| G) is a 5 
T , (1 Jo(wxo) } 


Sinusoidal 5 Jo(wao) | Gw) Pow — ma,) 
2 
Normal a exp (—aw) | G@) |6(@ — me,) (Ge) [1 — exp (—ow’)] 
: 2 2 . 2 
Flat 2n (stews) ones | G@) | E B pee 
T WXo 


WXo 


Double Spike 


— A) sin® (ec. oll 


- | G@) |?6@ — me,) 


| Gw) | u 
ne) 


ale 


iw) 


us 


T , 


On, = 


i — (VN), sally Sa, 9o° 


TABLE V(a)—Ranpom Putsr Duration Moputation, (PDM)—(ReEctTaneuLaR PULSE) 


Distribution 
Function 


Discrete Spectral Power Density Function 


P p(w) 


Continuous Spectral Power Density 
Function, 


P c(w) 


sin'( #1) 

i el 

Qa 2 wx 
0) s( 0 


Jat Bes 


2h” 
Tw 


we 


E — cos (wr) Jo(wap) 


Sinusoidal mit (er) 25 
io. wT wu 
: - 2a" (37) (22) | 
3 ey KO er 2h? ee 
en (7) 2y” E — cos (wr) ex ( ) 
2 Dy —_— 2 b D, p 
Normal i Oe ae ees. ( o° Jat — mw,) Tw 2 
‘ (es ; 22 2) 
5 — 2sin’ (2 exp ( ve) 
2E” 
Poe) E — cos (wz) = (wo) 
oP) | KEE - 2 | WLo Tw (wo) 
jet ear) eto alas 
Flat 5 HT 6(w mw,) 


Double Spike 


4E” 
To” 


ji? = VU, sail, Sey, occ 
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TABLE V(b)—Ranpom Putse Duration Mopuration, (PDM)—(Gaussian Pulse) 


Distribution Discrete Spectral Power Density Function, Continuous Spectral Power Density Function, 
Function 
Py(w) Pow) 
2 PA 72 
oO 0W 
Soe : - E’rr {1 2g r (: an 4 In ) ow ( —w ) 
Spuelieed ‘eat? | 4T In 2 ww? \*2 EO Sms Oar 
T 4in2(ow + 8 In 2 Sar aes 
ormal % 
Pe ne, . 
* ex Boy po aL OG: Si) 7 
BE ORE a ayo nas oe) 
ie 8 5 2 
E’x 2 2 2 2 
Tea 2 exp {—B (7 + 2)}{7 sinh (26°72) 
0 
Qn | B’r16 In 2 a eee 7 
0 
lat 
- sinh” (2A) Note — mw,) ae exp {—b'(r° + 2 )} sinh’ (07 = 
8 In 2 2 b°2x, p 0 0 
where b° aah a 9 
Qn | rH =) Ee ( 7 ) | ( =) 
T° F In sl an (= In 2 FONG ho) te rei 
Double Spike a eave 
2 2 
= (1 As) a exp =e slits 5(w— mw,) CK! = in 2) 


iw) 


Tv 


T° fi = \W), sell, Say, 


Oy = 


TABLE V(c)—Ranpom Putse Duration Mopunation, (PDM)—(Error Funcrion Puss) 


Distribution Discrete Spectral Power Density Function, Continuous Spectral Power Density Function, 
Function 
P p(w) PA) 
E | = 
A x(a — cos (2aR) exp (5) 
i qh! el Ab? 2\172 
Normal = 2) exp ie 2b'r + ao $ *) ie — mw,) (1 + a) 
| Peete 1 + 2928? : in? (aQ) (- A ee 
‘1 sin (aQ) exp eae 
1 + 20°b° 
Ae { a Vi 
——=— _ § erf[b V 2(7+ 20) ]— erf[b V 2(7 — a) ] 
eee ee Beal 0 
me | [ Mis a] 
& + ™ exp ($$)| Re en b Geran) ar 
A’m exp (ee a,) | 
2a 2b° . 7 ne 
1 T 160222 {im erf (ee bag a) — Reerf Be +x) + 5 x) ]} 


— Im erf (60 + bay + oF d(@ — muw,) 


a'r exp (53 


eT e? ae era erf (60 — ba + 


‘ 
2b 


— Im erf (60 + bay + a 
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TABLE V(c)—Ranpom Putse Duration Moputation, (PDM)—(lrror Funcrion Putse)—Concluded 


Discrete Spectral Power Density Function, 


Continuous Spectral Power Density Function, 


Function 
Pp) Pc) 
Qn A® ; 2 A’ A, — Ai) ,. 2: 
Double Spike ape | Ar sin (aa) exp (— 6'2) Wo y {sin (aa) exp (— b’2)) 
+ (1 — A,) sin (ar) exp (— b’x3)}?6(@ — mw,) — sin (ax) exp (— b’23)}? 
2B @ 2 aw a 2 
.=—, m= 0, £1, +2,--- =) =) = : = SASS 
@, m sell, ae A 3 a 9 b ee GY b 
: Sop ) ieee jb agen ae 
On can, IN! = ox ( aa) = ae = Se = 5 
ee ho Ry ok oe eaiee) og ee bg) ee ee 
TABLE VI—Ranpvom SineLteE Epes Mopunation, (SEM)—(ReEcTANGULAR PULSE) 
Pe Nate Discrete Spectral Power Density Function, Continuous Spectral Power Density Function, 
P p(w) P-(w) 
oer 2a BE : EY 
Sinusoidal 73 [1 + Joao) — 2 cos (wr) Jo(wao)]d@ — mw,) Te [1 — J5(w2o)] 
27K” ae 
a E + exp (—ocw) 
Tw BE 
Normal ae —-5 [1 — exp (-—ow)] 
=G06 Ta 
— 2 cos (wz) exp ( 5 ) Ja — mw,) 
9 2 1 2 : 
zac E + sin” (wo) — 2 cos (w7) 
To (wo) 2 eee 
E sin’ (wo) 
Flat ; —,) 4, —- 
_ sin lke. Ee Tw (wo) 
(w2o) 
4 E” : T 
Tp 1— A, + A; — A, cos Ot + 5 
Double Spike OR” 
— (1 — A,) cos (of i *}) Ta A, — A,)[1 — cos'@{x, — 2z,})] 


bo 


us 


T? i) = Wp sell, se, coc 


(Oy, = 


AY 


+ A,(1 — A,) cos ({x, — 22}) }6@ — me,) 
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A Note on the Sampling ‘Theorem 


L. Jo rOGRLY 


Summary—The human operator often perceives rate as well as 
amplitude information in sampling various displayed continuous 
parameters. It is therefore necessary to extend the Sampling 
Theorem to allow the analysis of certain man-machine relations. 
The result is stated and the required mathematics included in the 
Appendix. Certain distinct problem areas where this extension can 
be employed fruitfully are indicated. 


HE AIRCRAFT instrument — display-to-pilot 
"| ees channel normally operates with 

the sequential scanning of the various individual 
indicators, each concerned with a different variable. It is 
because of this modus operandi that it is necessary to 
apply the Sampling Theorem when taking a theoretical 
approach toward the analysis and design of aircraft 
instrument displays. To relate the required human 
operation to measured human capabilities, it is essential 
to include the fact that the human operator often observes 
derivative as well as function amplitude value in each 
sampling. For example, in the case of pointer-on-scale 
displays, he may estimate the pointer position (which 
corresponds to function amplitude), the rate and possibly 
even some information concerning the acceleration of the 
pointer (corresponding to first- and second-time deriva- 
tives). Thus, it becomes necessary to extend the Sampling 
‘Theorem to determine the periodic sampling interval 
required to fully represent the transmitted frequency 
limited message when the instantaneous sampling includes 
the function amplitude as well as derivative values. 

From the Appendix, it may be seen that when the first 
derivative, f(t), (n = --- — 2, — 1,0, + 1, + 2, ---) 
alone is added to the function amplitude sample informa- 
tion, f(t,), @ surprisingly simple result is obtained, 
T = 1/W; that is, twice the interval required when f (¢) 
alone is observed. The addition of each succeeding de- 
tivative allows the time interval between samples to 
become larger, according to T = (k + 1)/2W, where k is 
the order of the highest derivative when all lower-ordered 
derivatives are observed in each sample. 

Further, it may be seen that 7’ is a discrete increasing 
function of the number, M, of observed derivates. It 
should here be pointed out that the word “number’’ is 
correct, and is a true generalization of the k + 1 numerator 
when all & lower-ordered derivatives are observed. Thus, 
if the observer determines f(¢,) and f’’(¢,) only, the 
proper T is still 1/W. In general, the Extended Sampling 
Theorem may be stated as follows: 


If a function f(t) contains no frequency higher than W 
cycles per second, it is determined by giving M function 
derivate values at each of a series of points extending 
throughout the time domain, sampling interval T= M/2W 
being the time interval between instantaneous observations. 


+ Stavid Engineering, Inc., Plainfield, N. J. 


It is important to note that this result is not in conflict 
with the general statement that 2WT7 independent sample 
values are required to specify a function of duration 7 
and bandwidth W. It merely indicates various ways in 
which these independent samples may be observed. 

With this extended theorem established, various aspects 
of the aircraft instrumentation study program become 
accessible, and may be analyzed. From knowledge of the 
engineering aspects of flight and empirical data, it is 
possible to estimate the highest frequency component in 
the displayed parameter functions. The next step is to 
combine this data with the results of psychometric 
studies which approximately determine the human 
operator sampling process characteristics. An experimental 
study of human smoothing should reveal an estimate of 
the resultant observed signal-to-noise ratio, S/N, as a 
function of the received S/N during each sample. Since 
the observed S/N of higher ordered derivatives is lower 
due to judgment error “noise,” there should be some 
optimal compromise in the number of derivatives observed 
which will result in the highest S/N for the sampled and 
smoothed resultant received message. 

It is expected that further use will be made of this tool 
in human engineering analysis with regard to high per- 
formance aircraft flight control, ground vehicle control, 
and human operator target tracking. Careful consideration 
should also yield possible useful applications of the Ex- 
tended Sampling Theorem in the fields of air traffic control 
simulation, data link computation, and telemetry. 


APPENDIX 


Consider f(t) to be a function which admits of Fourier 
representation,’ then 


i) = x | slode** de. (1) 


If the frequency spectrum, s(w), of this function is bounded 
such that — W < w < + W, then 


1 +20 W 


f()= os s(wye’®’ dw. (2) 


—20° W 

The Sampling Theorem is derived from the fact that 
periodic sampling of this function, f(d), at the points 
nT, where T = 1/2W, produces the sequence of Fourier 
co-efficients f(n/2W), where n = --- — 2, — 1,0, + 1, 
+ 2, --- . Since s(w) is nonzero in a finite interval, this 
sequence, f(n/2W), determines s(w) and thus /f(é). 


1The function f(t) may have only a finite number of points of 
discontinuity and a finite number of maxima and minima in any 
finite interval and 


+o 


[ls lar<e. 
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Suppose the sampling interval is 7 = 1/W; then the 


sequence of Fourier coefficients becomes 


n il +2rw ; ; 
(2) = “ioe / Saar ida. 
J—-27W 


By a linear transformation of the sectionalized integral 
(folding), the following may be accomplished: 


1 WwW 27+ Ww 
= 9 : / s(w Ye? eae w + ie 
“7 0 


9 ane 


+ See ae dw ate 


J—x Ww 


(3) 


eer es dos! 


Soca ae tut} (4) 
J—-Q27 W 

Transform the second and fourth integrals according to 
w = 2nW + w and w = w — 2rW respectively then 
rearrange to give 


aw 
== { [ Be) + se — 2eW)]e"* de 


mis 


Since s(w) is symmetric with respect to the origin, so also 
are both bracketed quantities in the integrand. 

Suppose that the function derivative is also given, then, 
assuming it to be possible to differentiate under the 
integral sign,” 


[s(w) + sw + 2nW) Je"? dh (5) 


Lonl tees 
/ been Diner, inet 
1) = 5 | josleel** de, (6) 
If the sampling is performed in a similar manner, at 
t = n/W, and the frequency spectrum remains limited to 
the same range — W < w < + W, then as in (5) 


d. le [ws(w) 


+ @ — 2rW)sw — 2rW) Je"? dw 


4. [ws(w) + (w + 27W)s(w + 27 W) “ap. (7) 


= W 


The sequence f(n/W) and f’(n/W) give the simultaneous 
equations 


= = Y 7 
ond Qo(w) s(w) + s(w 27W) | (8) 
a(w) = ws(w) + @w — 24W)s( — 2rW) 
in the positive region of w, where a, (k = 0, 1) is the 


spectrum determined by the discrete set of points if the 
sampling length is taken as fundamental frequency, i.e. 


7Ww 


alco ie, 


J—T7W 


+k 
Si 


AS es 


s(w) can be determined from (8), thus f(f) can be de- 
termined. 


(9) 


2 A sufficient condition for this is that this integral, 


i. jws(w)e’ day 


converges uniformly and that the original integral in equation also 
converges (which has already been assumed). 


TRANSACTIONS—INFORMATION THEORY 


March 


In a similar manner, if 7 = M/2W where M =k + 1 
and k is the highest order of the complete derivative set 
which is taken, then 


font) = (2M) = b 


Qn r=0 


M-1 (27 W(rt+1))/M 
i (nM /2W) w 
Se / s(w)e’” des 
2 


J27Wr/M 
247 Wr/M 
+f s(w)e 7 (nM/2W)w da \ (10) 
{27 W(r+1)}/M 
where r = 0, 1, 2, --- (Wf — 1) and 
4 nM) we “he M-1 ‘ieee a “( = nV), 
I ee = Oe 22 lly ee emer 


.& 2rWr j(nM/2W) 
a( nro Je des 


"ey s(« ot a), i(mM/2W)0 gy 


(11) 


Each sequence {“(nM/2W) determines a function a,(w) 
defined by 


k a oak ae lend i(nM/2W) wo 
at DP eR: ae 
such that — 24W/M < wo < + 27W/M. A set of simul- 


taneous equations may be set up from the general ex- 
pression, 


0 


_ (yr {o x 


—27W/M 


(12) 


fora: > 0 
a,(w) = > (1 (o ee 2aWr) a ze) ag 
where k andr = 0, 1, 2, --- . (M — 1), that is 
aw) = sw) + (=o = es si a(« ant 
st) Ge iy, = ew) a 4 za) 
+(6- 47) e- 4p) 
Pt (y(n PRE) fs = 2a) 


A unique solution exists for s(@ — 2rWr/M) in terms 
of a,(w) if the determinant of the coefficients is not zero, 
which then determines f (¢). 

If the derivatives are not observed at the same time 
instants, it may easily be seen that this makes no change 
in the above derivation, since the Shifting Theorem allows 
the time shift to be expressed as a single exponential factor 
in the frequency domain 

vighle gy mal ef *s(u)el®* des. (15) 
hee 
The constructed set of M — 1 simultaneous equations 
will, in general, still be solvable under this change. 
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Statistical Calculation of Word Entropies 
for Four Western Languages 


G. A. BARNARD, IIIt 


Summary—Using a modified version of Shannon’s method,! 
-omparative figures for the word-letter entropies of printed English, 
‘rench, German, and Spanish are obtained and the method de- 
scribed. 


REVIEW OF SHANNON’S Meruop 
We’ his now universal definition of entropy,” 


Shannon has shown how the character entropy 

of a printed language can be calculated by 
leducting the asymptotic value approached by a series 
of approximations to the entropy. These approximations 
wre made by solving for the entropy in each case, using 
successively a conditional probability involving one more 
etter of preceding text. That is to say, 


3 p(b; , 7) logs p,(9) 


— 2) pb: , j) loge p(b: , 7) + D2 p(b,) loge p(b,); 


vhere Fy is the entropy calculation per letter for the Nth 
etter (based on conditional probabilities through (N — 1) 
etters), b; is a block of (NV — 1) letters, 7 is the Nth letter, 
und p,,;(j) is the probability of the Nth letter, 7, with 
-onditional probability dependency on the contents of 6; . 

Thus Fy is a measure of the average information 
-ontent of the Nth letter in a text when the (NV — 1) 
yreceding letters, with their statistical probabilities, 
wre all known and thus, theoretically, the entropy, 
PR ely 5 

For any language using the Roman alphabet, consisting 
of twenty-six letters, not considering punctuation or 
paces, 

a= oe 268 9457 0» 


For English, Shannon completed statistical calculations 
vf entropy for (b; + 7) equaling from one to three; L.e., 
‘, , Ff, , F; . These are the values for the computation of 
vhich frequency tables are available.’ Tables are also 
wailable for English word frequencies;’ Shannon used 
hese, also, to obtain a fourth value, F’,.,4, or /w , of 
ntropy. He presented arguments for showing ’y = Fg , 
Ithough average word length, for English, is 4.5 letters. 


+ Computer Lab., Engrg. Div., Stanford Res. Inst., Stanford, 
Yalif. 

1C. Shannon, “Prediction and entropy of printed English,” Bell 
ys. Tech. Jour., vol. 30, pp. 50-64; January, 1951. 

2C. Shannon, “A mathematical theory of communication,” 
sell Sys. Tech. Jour., vol. 27, pp. 379-423; July, 1948, pp. 623-656; 
etober, 1948. 

3F. Pratt, “Secret and Urgent,’ Blue Ribbon Books, New 
ork, N. Y.; 1942. 

4G. Dewey, “Relativ Frequency of English Speech Sounds,” 
[arvard University Press, Cambridge, Mass.; 1923. 


Rather than attempt the colossal task of compiling 
tables for further values, Shannon used an English speaking 
subject as an experimental “predictor”? to obtain values 
of F, through Fy; , and Fio. . Armed with probability 
tables for two and three letter combinations, plus his 
life’s experience with English, its grammar and word 
order, this person made calculated guesses of the “next 
letter,” 7, in the different length phrases. His correct and 
incorrect guessing helped furnish data from which Shannon 
could show enough values of /'y, to get a good asymptotic 
approximation for H. 

The word list which Shannon used for the calculations 
to obtain Fy is a list’ of 1,027 words composed from 
sampling 100,000 words of printed English text. This 
letter entropy per word was evaluated as 


M 


Fw = — 2) pp logs pp (1) 
where - 
(= : (2) 
and 
S Pr = 1, (3) 


for consistency, assuming that the list ends at n equal to 
a value such that the total of all probabilities is unity. 
Shannon then plotted his word-rank versus word- 
frequency on log-log paper and by inspection found 
k=0O.1. 
Examination of his results shows his method to have 
been the solving for M in (8) by the approximation 


M J M 
epee pee / p, dn = 1. (4) 
n=1 n=1 JD) 


Then he appears to have solved for the letter entropy per 
word by 


1 M 
Fy = = (2 > p, logs p.): 
n=1 


approximated as 


1 Uf M 
Py = = be Dp ORs Dye i Pn logs Dn in| 
1] ak, _ &k ip eileeneemd | 

ate pe loge + [7 lows an 


a 1 n 


af 


k ene 


rT ib i n 


M 

athe 

—dn — 
n 


+ log, k / 


J.5 


" log, n 
k soar an| 
n 


JJ.5 
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J log, n 
= “4 ye Staten Uh 


1 n 


“ logs n 
+ Se 
ween 
and since, in (4), 


ay My), 
d A + ie 7 an = |. 
~! | to. , x > 18 log. n ot * logs n , an) |, 5) 
a 1 Jisio mu 


where a = 4.5 is the average number of letters per word 
for English and J = 1,027 is the list length. 

Except that it is an interesting figure to know, the 
direct solution for M could have been eliminated, since, 
in both (4) and (5), the integral portion of the solutions 
contains In M as the only remaining unknown. Thus (5) 
would be 


Py = 


J M 
an = "tog. k- >! ae 1 44269 ue an) | 


1 J.5 


since log, n = 1.44269 In n 


( J 
i ee ion 
a 1 l 


se {One hc Lh), 6) 


J . 
ae + k[ln M — In J.5] = 1 


n=1 


a 


and (4) would be 


or 


eee Ns dha 


J 


1 1 
mn oa 


1 


ip hae hoe (7) 


Shannon’s examination of his log-log plot gave him 
k = 0.1, the word list gave 
1027 7. 


~ = .78633; 


n=1 n 


and the expression 


can be computed from the word list data. 

What Shannon’s method amounts to, then, is, in (1), 
taking the sum of the actual experimental values in a 
100,000 word total count of (p, logs p,) out to a word-rank 
limited by the extent of the word list, then taking the 
sum of the mathematically computed values of this same 
expression out to a value of n, word-rank, where the total 
probability of all words equals unity; p, being figured as 
equal to k/n, and k estimated from a graphical plot of the 
word list probabilities versus ranks. 
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CoMPARATIVE Mrruop OF COMPUTATION WITH 
Rootrep Worp Lists 


Should equivalent word-frequency lists be available for 
other languages, particularly ones using the Roman 
alphabet (or any other of 26 letters), a valuable com- 
parison could be made of the entropies (and thus re- 
dundancies) of these languages. Knowledge of which 
languages had the highest entropies (or lowest redundan- 
cies) could be a guide towards shortening message lengths 
for given information content or towards synthesis of the 
most efficient machine language for translation devices 
or for computing machines. 

The list used by Shannon for English was one making 
full distinction between all words of different spelling. For 
example, the words “I, I’d, I'll, I’m, would, will, am” 
are listed as seven discrete words, the listing for the word 
“Would” not containing a count of its apostrophized 
counterpart in “T’d.” 

For this paper, investigation has been made of French, 
German, and Spanish, as these and English are the four 
principal Janguages of the western (Roman alphabetized) 
world today. However, word frequency lists of the same 
type as the English one above are not available in these 
three languages; rather, there are lists of words grouped 
according to roots,’ for example, all conjugated forms of a 
verb being listed under the infinitive as one composite 
word listing. Such a list is also available for English. The 
original reason for compiling word lists of any kind was 
for pedagogical purposes, and, although the pure lists must 
have existed first in the compilation process, they were 
not published, since the rooted list is the one most useful 
for the teaching of languages. 

Investigation of the problem of compiling the desired 
lists showed that the task for only one language would be 
out of the question for this paper. In compiling his list for 
English, Dewey says, “The word counting stage was 
expedited and checkt as far as possible by mechanical and 
other aids, including specially printed record cards, 
Veeder counters, Rand card holders, Smith steel signals, 
and a special card ledger desk. This stage, with fonetic 
transcription on the original cards which accompanied it 
required in all about 1200 hours.’’* This is the same as 
five months at eight hours per day, seven days per week. 
The same work could now be done on an IBM punched- 
card calculator for an estimated total cost of $500 
(materials, operator labor, machine rental) and a minimum 
time of one month (eight hour days, five days a week), for 
a total count of 100,000 words. 

It was decided to make a first step in the comparison 
process by using the rooted lists available, thereby getting 
a rough comparison of the entropies and pointing out a 
method of approach for better comparison in later work. 
The basic assumption necessary in accepting these lists as 


> B. Q. Morgan, “German Pay Word Book,” The Macmil- 
lan Company, New York, N. Y.; 1928. 
G. E. V anderBeke, ‘ ‘Fr ench Ww. ord Book,” The Macmillan Com- 
pany, New York, N. 3 1929. 
M. EB. Buchanan, aa Graded Spanish Word Book,” The Uni- 
versity of Toronto Press, Toronto, Canada; 1927. 
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the bases of comparison is that they have each covered the 
same general fields of literature with equal proportioning 
between the various literature fields. And equally im- 
portant is that the combining into root groups is done in 
an equivalent manner with each list. This latter is un- 
doubtedly the less certain for these lists. 

The results, based on a pure 26-letter alphabet, are as 
follows: 


English French German Spanish 
Fo 4.70 4.70 4.70 4.70 
F, 4.124 3.984 4.095 4.015 
Fy 1.648 3.02 1.08 1.97 
(w = 4.5)3 (w = 4.84)3 (w = 5.92)3 (w = 4.96)3 


Tables® are available for the single letter frequencies of 
the four languages, so that the computations of /; come 
simply from 

26 
iP — — > 0; logs p; 5 
i=1 


The method used for solving for Fy is as follows: 


1. Plot a graph of word frequency (probability) versus 
word-rank on log-log paper. 

2. From the graph, determine k as defined in (2). (This 
mathematical relationship between p, and n is used, 
based on a statement by Zipf* that it is a good approxima- 
tion for many languages.) The complete graph should not 
be used to estimate k since often it is much more nearly 
approaching its asymptote after a rank, n, of several 
hundred. In this paper the range from n = 200 ton = 800 
was used, principally because the four lists to be compared 
contained specific data through this range. The range for 
estimation should not be too near the low-p, end of the 
graph either, since the accuracy of the probability figure 
gets lower as n increases, for any given total word count. 
For example, the probability figure of 0.0731 for the 
English word ‘‘the’’ (7,310 occurrences out of 100,000) is 
much more reliable than the figure 0.00011 for the word 
‘worse’ (11 occurrences out of 100,000). That is to say, 
the smaller the number of occurrences within a given text 
sampling, the more the probability figure depends upon 
the character of the particular text samples chosen for the 
test. The word “‘worse’’ could easily have occurred only 5 
or 6 times out of 100,000 if a slightly different group of 
texts had been chosen for the tests—thus getting its 
probability figure to be one-half what it was with 11 
occurrences. This is illustrated by noting that at their far 
end two of the graphs plotted herein begin to drop down 
and k gets smaller. 

3. Having obtained k, we now dispense with the lists 
and curves, some of which, incidentally, have no data for 
the first several word-ranks, and assume a pure hyperbolic 
curve beginning at n = 1 and ending at the value of n 
where Day; p, = 1. Thus we use (6) and (7) derived above, 
but with mathematically determined values, rather than 
empirical ones, for the expression >> k/n and > In n/n; 


6G. K. Zipf, “Human Behavior and the Principle of Least [ffort,” 
Addison-Wesley Press; 1949. 
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the upper limit of these sums can be as high as needed to 
stay within any desired degree of accuracy when the 
integrating approximation is used for the end portion of 
the evaluation. This limit can even be chosen as equal to 
M, no integration then being needed. For this paper, the 
limit was chosen arbitrarily as 100, giving us the ex- 
pressions for (6) and (7) as 


f = ie 
eel 
as = ele TA: 
bn I = 4 GSIOIUG 
ib peat 
nevi odie 
k 
Foes ‘ (— 1.44269 Ines ieeakeee ee e 3278). 


The principal problem was the obtaining of the figures 
for p, in each language. For English, this was easy since 
Dewey’s rooted list is tabulated by order of greatest 
occurrence and the number of occurrences out of 100,000 
are given for each word, so that p, = f,/100,000 = 
f. X 10°. These values were plotted directly for the 
points n, shown in Fig. 1. 
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Fig. 1—Probability—rank curve for English words (rooted data). 
800 


n=200 


For German, Morgan’s adaptation of Kaeding’s list 
eroups the rooted words into blocks lying between two 
frequency limits, giving us accurate frequency values for 
only first and last words of the block. These were taken 
from Kaeding’s total count of 10,910,777 (See Fig. 2). 

For French, VanderBeke gives figures for Henmon’s 
original list, based on a count of 400,000 through n = 63. 
For n > 63, VanderBeke’s count is 1,147,748. The list is 
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Fig. 2—Probability—rank curve for German words (rooted data). 
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k] = .157. 


n=200 


published alphabetically, so that the figures were obtained 
by sorting according to frequency as in Fig. 3. 

For Spanish, Buchanan published his frequency list in 
a modified form whereby he gives a weighted value to the 
frequency figure, the weighting depending upon how 
many of the different separate sample texts contained the 
word; his figure was (f/10 + 7) where f is the actual 
frequency of occurrence out of the count of 1,200,000 
words and r is ‘range,’ or the number of text samples 
containing the word. He also published an alphabetical 
list of his 6,513 different words, giving both f and r. 
Rather than rearrange this whole list to obtain the few 
points needed for the graph, in this paper the method 
used for determining p, was that of picking a word, n, 
in the weighted-frequency list and, from the alphabetical 
list, determining its f together with the /’s of the five 
words preceding it in the frequency list and the five words 
following it; that is, 


Buchanan made his list with the effort-saving assumption 
that words of ranks n = 1 through n = 189 were im- 
portant enough so that, for pedagogical purposes, they 
need not be counted. Thus Fig. 4 (next page) gives no data 
at all for the first part of the curve. 


COMPARISON oF THE Two MprrHops 


Had Shannon used the method herein described for 
solving for Fy , using, as he did, the nonrooted list for 
English, he would have used (8) with a = 4.5, k = .103 
as obtained from Fig. 5 (next page). This graph is plotted 
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Fig. 3—Probability—rank curve for French words (rooted data). 
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from the nonrooted word-frequency list. Then Fy would 
have been: 


1 


ican 7h 


E 1.44269 In .103 


721345 


iit $3278) 


“12661-7103 =F 


2.103. 


Shannon’s direct method obtained Fy = 2.62, the differ- 
ence being due chiefly to the use in this paper, in (6) and 
(7), of mathematically derived data out to only J = 100 
whereas Shannon used statistical data out to J = 1,027, 
and to a certain extent to the difference between the 
values of k. The smaller the k value, the larger the entropy. 


COMPARISON OF THE UsE or ROOTED VS 
Unrootep Data 


It is well to note at this point that Fy is greater for 
unrooted data than for rooted data. As we see above, 


1.648 
2.103 


135 
103. 


This is because, in (8), k being less than unity, the increase 
of the first and third terms with decreasing k is much 
greater than the decrease in the second term; hence the 
increase of Py . 

It seems reasonable to deduce the assumption that the 
same situation would be true in the other three languages. 
Certainly the average values of p, would increase for 
given values of n since the occurrence rate of a rooted 
word equals the sum of the occurrences of all words for 
which it is the composite. That is to say, since p, increases, 
k must increase for given n; p, & k/n [see (2)]. 
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Fig. 4—Probability—rank curve for Spanish words (rooted data). 
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Discussion oF RESULTS 


The first point to note, in the results, is the fact that 
the average word length bears directly upon the values of 
Fy . It would be expected, then, to find yw smaller for 
German than for the others since the German words are 
inordinately longer than in the other languages considered. 
The trend of smaller differences in word length between 
the other three languages themselves, however, does not 
follow this discussion for German. 

To account, then, for the other differences in Fy between 
the languages, our immediate conclusion is that French 
must have a smaller vocabulary for expressing the same 
ideas as the others, i.e.: it appears to have a few explicit 
words, in contrast to many different words in other 
languages, to express the same idea. This reasoning could 
apply to German, also, in addition to the actual word- 
length as discussed above; the deduction being that 
German has less explicit words and needs to combine 
several to express an idea. We know that this is actually 
the reason for the long word-length in German: its long 
words are almost always found to be several small words 
combined into one. A check on this idea, which is beyond 
the scope of this paper, would be to compare the total 
number of words needed by each of the four languages for 
translation from an identical passage written in a fifth 
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Fig. 5—Probability—rank curve for English words (nonrooted data). 
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language; for example, the New Testament of the Bible 
translated from Greek or Aramaic. 

For one lacking the conviction that French is the 
most explicit—or so much more so—of the four languages, 
the obvious step to take is to examine the validity of the 
two basic assumptions made above before accepting our 
data lists. Since we have found how much the value of 
F, depends on k, and how much k depends upon the way 
in which the rooted words are grouped together, (in effect. 
the larger the grouping together, the more explicit we are 
making the vocabulary) the more we realize how important 
it is for the root grouping to be equivalent in our data of 
the languages compared. Equivalence of literature fields 
covered is fairly easy to check, by examining the lists of 
the works from which the sample texts were taken. 

Shannon’s paper’ brings out the upper and lower bounds 
obtainable experimentally (as described above) for the 
various values of /'y, . His result for Fw in English, using 
nonrooted data, falls on the upper bound. The result for 
English found in this paper, using rooted data, falls near 
the lower bound. 
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On the Modulation Levels in a Frequency Multiplexed 
Communication System by Statistical Methods” 


R. L. BROCK} AND R. C. McCARTY} 


Summary—This paper presents a mathematical analysis with 
experimental verification of the distribution of the instantaneous 
voltage of a complex signal resulting from the combination at ran- 
dom of a small number n of sinusoidal oscillations. The resulting 
calculated distributions are plotted in the form of a set of probability 
curves for comparison with curves obtained by experiment. Further 
laboratory measurements in which the individual sinusoidal oscil- 
lators are frequency-modulated in a manner suitable for communicat- 
ing information in a binary form, yield substantially no change in the 
amplitude distribution as determined for the unmodulated oscillat- 
ors. Consequently, the results of the mathematical analysis may be 
applied in the determination of 1/, the degree to which each sub- 
carrier may amplitude modulate a final carrier in a FM-AM fre- 
quency multiplexed system. 1/ may be determined for any desired 
degree of overmodulation in excess of one per cent and for as many 
subcarriers as are required in the system. Modulation levels de- 
termined according to approximate methods and the method de- 
scribed here are tabulated and compared. 


INTRODUCTION 


ULTIPLEXED transmission of information 
IM using double and triple modulation systems 
has assumed increasing importance in recent 
years. Such systems may employ either time or frequency 
division. In frequency division, which is of interest here, 
individual subcarriers spaced in frequency may be 
amplitude-modulated (AM), frequency-modulated (FM), 
or single sideband-modulated (SS); groups of subcarriers 
may be combined to amplitude modulate or phase 
modulate a carrier. 

One of the more interesting problems arising in the 
design of a frequency multiplex transmission system in 
which the final carrier is amplitude-modulated is that of 
determining the allowable degree to which the individual 
subcarriers may amplitude modulate the final carrier. 
Since frequency modulation results in a varying frequency 
sine wave of constant amplitude, it will be convenient 
mathematically to consider an FM-AM frequency multi- 
plex system. That is, a system in which a group of equal 
amplitude, frequency-modulated subcarriers are linearly 
combined to form a complex signal that is used to ampli- 
tude modulate a higher frequency carrier. In order to 
insure a high signal-to-noise ratio for a particular sub- 
carrier signal, it is necessary to make M, the factor by 
which each subcarrier modulates the main carrier, as large 
as possible. The optimum choice for M will depend on the 
system distortion that can be tolerated due to over- 
modulation and the distribution of the instantaneous 
voltage of the complex signal resulting from the combi- 
nation of all subcarriers. It is the determination of the 
latter that is of primary interest here. 

* Presented at the Western Electronic Show and Convention, 


Los Angeles, Calif., August 25-27, 1954. ; ‘ 
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DETERMINATION OF MopULATION LEVELS 


In a recent paper Landon’ indicates that for an FM-AM 
system the modulation index M should be determined 


according to 
Ve Ch) 


(1/8n)"”” for large n, (2) 


n * for small n, 


M 


where 7 is the number of subcarriers and M is as previously 
defined. Choosing MW according to these relations, effec- 
tively assures that overmodulation will not occur. 

Kq. (1) is based on a linear summation of the individual 
subcarrier amplitudes. Eq. (2) is a result of the following 
reasoning: In another paper Landon* has shown that the 
distribution of voltage for fluctuation noise is normal and 
has a crest factor of about 4. That is, the ratio of the 
amplitude of the highest peaks to the rms value is 4. 
Peaks higher than this do occur but are highly improbable. 
If it is assumed that for large n the sum of » frequency- 
modulated equal amplitude subcarriers behaves like 
fluctuation noise, then the probability that the absolute 
value of the instantaneous voltage of the resultant signal 
exceeds y times the rms amplitude (standard deviation) is 


fo) 


P(y) = 2(2n)7” / ont de (3) 


y 


1/2 


The rms amplitude is (n/2)’~ times the peak amplitude 
E of a single subcarrier, and if M is chosen to give 100 per 
cent modulation when the instantaneous voltage is y 
times the rms value, then 


y(n/2)'"E = EB, , (4) 


where /#, is the peak amplitude of the unmodulated 
carrier. Since M is H/E, it follows that 


M=y7'(2/n)”. (5) 


If y is assigned the value 4, then (5) becomes (2), and 
from (3), which may be evaluated from tables of prob- 
ability functions, the probability that the instantaneous 
voltage of the complex signal will exceed 4 times the rms 
value is essentially zero (P = 0.000064). 

It is possible to choose y < 4 in which case (5) permits 
a larger modulation level and (3) gives the corresponding 
probability of overmodulating or, on the average, the 
percentage of time overmodulation occurs. The resulting 
overmodulation appears to the receiver as noise and the 
degree to which this is tolerable depends on the noise 


1V. D. Landon, “Theoretical analysis of various systems of 
multiplex transmission,” RCA Rev., vol. 9, pp. 287-351; June, 1948. 
°V. D. Landon, “The distribution of amplitude with time in 
fluctation noise,” Proc. I.R.E., vol. 29, pp. 50-55; February, 1941. 
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power present from other sources and the resultant degree 
of loss of information. 

As the modulation level M is increased, the signal-to- 
noise power ratio S/N for a given subcarrier will increase. 
Even for M sufficiently large to cause overmodulation on 
the peaks, the S/N should continue to increase with M 
until the resulting overmodulation noise becomes com- 
parable to the thermal noise of the receiver in its environ- 
ment. Beyond this the S/N may be expected to decrease 
with increasing M. It is apparent that allowing some 
overmodulation to occur can offer real gains in the single 
subcarrier S/N. 

Landon*® has suggested that the normal distribution 
which should be employed above is n = 8 to 12, but for 
smaller n the linear summation should be used. The 
use of linear summation (while perhaps suitable in some 
applications) is not based on knowledge regarding the 
distribution function for small n. The value of n below 
which the normal distribution no longer describes the 
situation accurately can only be determined by a com- 
parison with the true distributions. In what follows, the 
true distributions for small n will be approximated 
mathematically and verified experimentally. 


MATHEMATICAL ANALYSIS 


For the case of frequency modulation, where the 
sinusoids are of constant amplitude, the resultant in- 
stantaneous value of multiplexing » subcarrier signals 
simultaneously from mn independent sources may be 
expressed as 


E, = E >> cos (ot + ¢,) (6) 
k=1 
or 
Jin = IU} Ds cos 6, , (7) 
k=1 


where 6, = (w,¢ + @,). All values of the initial phase 
angle ¢, are equally probable in the interval — 7 < ¢ < 7m. 

The behavior of the instantaneous values LH, for a small 
number of subcarriers, (1.e.) m < 12, which is of particular 
interest here, can be readily determined by the methods 
of statistical analysis. 

To determine the statistical properties of EH, , it is 
necessary to examine the distribution of HL cos 6. @ in (7) 
assumes all values in the interval — « < @ < 7z, with equal 
probability. Thus it is seen that the absolute value of HL 
cos @ is distributed in the interval, 0 < | # cos @| < EB. 
If the individual sinusoids are combined at random, then 
E,, assumes a value at any time ¢, represented by the sum 
of n items drawn at random from a parent population 
whose distribution is that of EF cos 6. When 6 assumes 
values in the interval bounded by 0 and z, F cos @ takes 
all values in the interval whose limits are + HL, thus it is 
unnecessary to consider the entire range of variation of 0. 


3V.D. Landon, ‘‘Theoretical analysis of various systems of multi- 
plex transmission,” RCA Review, vol. 9, pp. 433-482; September, 
1948. 


Brock and McCarty: Modulation Levels in a Frequency Multiplexed Communication System 59 


The probability of any value of 6 in its range 0 < 6 < 7, 
is d0/. Thus, for the parent population, we consider the 
general expression y = E cos 6, which yields, 


dé dy 
a(E sin 6) is) 


The density function f(@) for EH cos 6 is then 


1 
(FE sin 6) 


f(@) = 
and therefore 


1 


WC) (10) 
the minus sign being neglected as all + values of y are 
equally probable. 

The representation of (10) by one of the types of density 
functions in the system developed by Karl Pearson,* 
provides a unique method of treating the statistical 
properties of the instantaneous values of y, and con- 
sequently E,, , when applied in conjunction with the works 
of Tchebycheff and Church. 

In the Pearson system, only the first four moments are 
required to determine the nature of any distribution, with 
the exception of some degenerate types which may be 
determined by a smaller number. 

Kendall’ has shown that a distribution is uniquely 
determined by its moments if lim,,.... v,’"/n is finite. As a 
corollary, a distribution is unique when its range is finite. 
For example, given the fact that the range of the dis- 
tribution is from a to 6 and taking the origin at x = a, 
with the interval b — a = ¢, 


Vs i RCL) Oe Ce lay) 


therefore 


(tin) eae Gin en Se 


and thus lim,.. v/"/n = 0 where »y, is the absolute nth 


moment. The range of the distribution whose density 
function is f(y) is obviously finite and therefore the con- 
ditions for uniqueness are fulfilled. The first four moments 
of f(y) are determined by the evaluation of 


(12) 


+E 
m= {WI dy (13) 
and 
+E 
Un = i (y — u1)"fy) dy, (14) 
= 95) 
which yield 
Ee 3E* : 
il = 0, Lo 2? Lee 0, by = Ee (15) 


The second, third and fourth moments were taken about 
the mean, u’ . The moments of odd order vanish because 


4M. G. Kendall, “The Advanced Theory of Statistics,” Hafner 
Publishing Co., New York, N. Y., vol. 1, pp. 187-148; 1952. 
5 [bid., pp. 108-109. 
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the distribution is symmetrical about the mean. The 
parameters of the distribution 8, and 8, are defined in 
terms of the three moments pu. to uy by 


Bo = ps/mo . (16) 


Evaluation of (16) gives 6, = 0, and 6, = 3/2. Pearson’s 
criterion AK, for determining the type of density function 
in terms of its parameters, is given by 


i; S48 
8, = Ms/ Me , 


ah eS Grea: y 
* ~ 4B, — 38,)(28, — 36: = 6) ee 


which yields K = 0 for f(y). This, when considered in con- 
junction with 6, and @, , indicates that f(y) corresponds to 
the Type II density function of the Pearson system. 

From a method provided by Tchebycheff, and discussed 
by Church® and Cramer,’ the distribution of the means of 
samples of a small number of items n, selected at random 
from a parent population can be determined with reason- 
able accuracy, even for populations whose distributions 
are slightly skew, by relating the moments and parameters 
of the Aistaeution of the means of the samples to those of 
the sampled population in the following manner: 


(a) M, = py & B, -4 
(b) M, == 
; () B, = 3 + @—4 
(c) M; = m (18) 


| 


(d)M, = 3+ Ee Dh 2] 


where M, , M,,M,;,M,, B, and B, are the moments and 
parameters of the distribution of means of samples of n 
and uw; , M2 , Ms , Ha » 81 , and B, represent these quantities 
for the parent population. Church has also shown that if 
the parent population is distributed as a Type I, II, III, 
IV or VII distribution of the Pearson system, with param- 
eters of moderate size, the distribution of the means of 
the samples and parent population will be of the same type. 

Having previously shown that the density function 
f(y) determined from the E cos @ population corresponds 
to the Type II density function of the Pearson system, it 
follows from Church’s work that the distribution of the 
means of samples from such a population corresponds to 
the Type II distribution. 

Once again we consider (7), where the instantaneous 
value H,, at any time ¢ is 


E, = E > cos 6, . 
k=1 
The mathematical expectation or mean of cos @ is then 


E, 
= =a cos 6, = a7 (19) 


6A. EK. R. Church, Biometrika, vol. 18, p. 321; 1926. 
7H. Cramer, ‘““Mathematical Method of Statistics,’ Princeton 
University Press, Princeton, N. J., p. 345; 1950. 
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Thus, letting « = H,/nH, the density function of the 
distribution of the means of samples of cos @ is 


2 


(i=), st ae CO 


i 
aB(1/2, m + 1) 


f(x) = 


which is the Pearson Type II density function, where 


B 2B,M, 


—— eS le 
Cane gl 2 Moa). 
(SBs — 9) ma 

= — 50 < Ob hy y 
33 — B,)’ 1505 it, 2 eo (21) 


for 2 <n < 12. B(1/2, m + 1) represents the complete 
Beta function. The distribution function F(x) is given as 


Mean 2 ‘ tes 2 DINO. st 
BO ear Fatt (1 — P/a?)" dt. (22) 
Letting u = t/a’ 
(x1/a)? ees 
Fa) = BG oie arr i = w* du 
= Tapaye (y/2; m ae Ne (23) 


the incomplete Beta function. 

Thus, the probability of exceeding any particular value 
of the mean of cos @is 1 — F(a). 

From (23), then 

a E el Bay Ne 0 | (24) 
where p = 1/2,g =m-+1. 

This expression is readily evaluated since the values 
of the function, [¢:/a): (p, g) have been compiled and 
published in tabular form.” The resulting probability 
curves are shown in Fig. 2. 


EXPERIMENTAL PROCEDURE 


Laboratory equipment suitable for measuring the 
amplitude distribution of complex signals resulting from 
combination of n = 2, 3, 4, 5, 6 sine waves of different 
audio frequencies has been developed for the purpose of 
checking the validity of the mathematical analysis. Fig. 1 
(opposite) is a block diagram of the test equipment. 
The assumptions in the mathematical analysis regarding 
randomness of phase should be fulfilled experimentally 
since the sinusoidal signals are derived from independent 
sources and all values of initial phase angle are equally 
probable for each oscillator. 

To correlate the experimental considerations with the 
theoretical results it is necessary to determine experi- 
mentally the probability that the instantaneous voltage 
of the complex signal exceeds some arbitrary value R. 
This may be reduced to a measurement of the percentage 
of time the instantaneous voltage exceeds R, providing 
the measurements are continued for a time long enough 
to give statistical meaning to the data. 


* Karl Pearson, ‘Tables of the Incomplete Beta Function,” Uni- 
versity Press; 1934. 
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Referring to Fig: 1, the equal amplitude outputs of the 
desired number of oscillators are linearly summed in the 
mixer to give a complex resultant signal which is amplified 
and applied to the Schmitt trigger input. The Schmitt 
trigger is adjusted to operate at the input voltage level R 
and to remain in operation until the input signal drops 
below &. During this time a positive de square wave 
available at the output of the Schmitt trigger enables the 
gate following, thus allowing the timing signals from the 
standard frequency source to pass through the gate and to 
be counted by the counter. 


OSCILLATOR 
|| 
OSCILLATOR 
Fo e 
Peper J. 
#3 
OSCILLATOR 
r4 
OSCILLATOR 
5 
ane PA 
#6 


SINUSOIDAL 
OSCILLATOR 


STANDARD 

FREQUENCY 
TIMER 

SOO KC/ SEG. 


DIFFERENTIATOR, 
AMPLIFIER, 


CATHODE FOLLOWER 


TO ELECTRONIC 
COUNTER 


Fig. 1—Block diagram of equipment employed for determination of 
the distribution of amplitude with time of the complex wave resulting 
from linear summation of a number of sinusoidal signals. 


The percentage of time the instantaneous voltage of the 
resultant signal exceeds R may be determined by the 
ratio of the number of counts observed to the total number 
of counts possible during the time of observation. If the 
experiment is repeated for different values of FR, it is then 
possible to plot a curve of the percentage of time the 
instantaneous voltage exceeds R as a function of R. 

The timer and counter must each be capable of operation 
at rates sufficiently high to give an accurate measure of 
the duration of each excursion of the complex signal 
above the value R. This is critical when RF is large and thus 
the excursions above FR necessarily of short duration. 

The measurements were made in the above described 
manner and the results are plotted in Fig. 2. In determin- 
ing the percentage of time a particular value of Ff is ex- 
ceeded, the mean value of fifteen observations over 
periods of one second each was used. 

It is of interest to note that frequency modulating the 
individual oscillators in a manner suitable for communi- 
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cating information in a binary form did not yield results 
significantly different from those plotted in Fig. 2. Thus 
it appears valid to employ the results of this analysis in 
determining allowable modulation levels for an FM-AM 
transmission system. 


PROBABILITY OF EXCEEDING THE MEAN 
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Fig. 2—Distribution curves of n sinusoids combined at random. 


Theoretical 
Experimental 


DISCUSSION OF RESULTS 


The methods of Pearson and Tchebycheff, in addition 
to other well-known methods, were applied to a similar 
theoretical analysis by Margaret Slack.” However, it was 
stated in Miss Slack’s treatment that representation of 


9M. Slack, Jour. IEE, vo). 93, p. 76; 1946. 
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TABLE I 


Mopvutation INpIces CaLcuLATED AccORDING TO THE VARIOUS Mrruops DiscusseD 


(A) (B) (C) (D) (E) (F) (G) (H) 
| Assumed Normal Distribution, M@ = y1(2/n)}”? Distributions of Fig. 2, M = 1/xn 
Linear | No Over- | Overmodula- | Overmodula- | Overmodula- | Overmodula- | Overmodula- | Overmodula- 
Summation modulation tion Time tion Time tion Time tion Time tion Time tion Time 

(Landon) | (Landon RMS) 1% 6% 15% 1% 6% 15% 

n M=1/n | y=4 y = 2.575 pee 1.88 y = 1.44 
2 | ay 249 388 | 532 694 = 556 676 
3 330 204 Olt 435 .568 a 441 556 
4 . 250, 177 .275 O16 491 . 296 382 481 
5 | . 200 158 245 336 .438 . 260 341 435 
6 NSE 144 224 307 401 .235 309 392 
8 125 ml 194 . 266 347 . 200 . 266 344 
10 100 lL? 174 238 dll 178 . 238 308 
12 083 . 102 159 PANG 284 162 219 282 


the true distributions by distributions resulting from 
evaluation of the Type II density function of the Pearson 
system was inaccurate for the cases of n = 3 and n = 4. 
A comparison of the results obtained here, using the 
Pearson Type II density function, with those obtained by 
Miss Slack employing another method, indicates that such 
a conclusion is unwarranted. 

In addition, it was found that Miss Slack’s values, 
obtained by the use of the Pearson system for 6 < n < 10; 
appear to be somewhat higher throughout the entire 
range of the probability domain than those obtained in 
this analysis. A similar disagreement with the results for 
the region of maximum amplitude (small probabilities) 
obtained by Miss Slack from the normal distribution for 
the case where n = 10, has been mentioned in an analysis 
performed by Bennett,’ utilizing a-different method. 

It is also of interest to note that the values obtained in 
this analysis by use of the Pearson system for the case of 
n = 10, are in good agreement with those obtained by 
Bennett. 

The theoretical and experimental distributions as 
plotted in Fig. 2 are in close agreement for probabilities 
greater than 0.01, with the exception of n = 2. Since 
experiment shows the distributions to be essentially un- 
changed when the sinusoids are frequency modulated, it is 
valid to determine the allowable degree M to which each 
subcarrier may amplitude modulate the final carrier in 
an FM-AM frequency multiplexed system according to 
these distributions. The data in Table I compare modula- 
tion levels calculated according to the various methods dis- 
cussed, for different degrees of overmodulation. 

1 W. R. Bennett, Quart. Appl. Math., vol. 5, p. 385; 1948. 


From Table I it is observed that for n > 8 the modu- 
lation levels M determined with the normal distribution 
and the distributions of this paper are in two figure 
agreement for the three overmodulation times con- 
sidered. Thus in practice the normal distribution may 
be assumed, when overmodulation is allowed, in the 
determination of M for n > 8. If overmodulation is 
not desired, then the normal distribution may still be 
assumed, as in column (B) for n > 8, but linear summation 
as in column (A) must be employed for n < 8. Use of 
the linear summation of column (A) for n > 8 would 
also assure no overmodulation but would result in ineffi- 
cient use of the system because of unnecessarily low 
modulation levels. 

For n < 8, if overmodulation time is not to exceed one 
per cent, then comparison of columns (C) and (F) shows 
the error in assuming the normal distribution to be 
appreciable for small n but to decrease with increasing n. 
If larger overmodulation times are to be tolerated, then a 
comparison between columns (D) and (G) and (E) and 
(H) shows the error to be somewhat smaller for all n < 8, 
but again tending to be greatest for small n. 

That approximating with a normal distribution leads 
to errors generally dependent on the overmodulation 
times allowed, is a consequence of the fact that the dis- 
tributions found here are approximated by the normal 
distribution better for some probability regions than for 
others, the regions of low probability showing the greatest 
deviation. Because of this and since the normal distribu- 
tion becomes generally a poorer fit as n decreases below 8, 
it is advisable to determine allowable values of M according 
to the distributions presented here for n < 8. 
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The calculations determining the data in Table I for the 
cases where overmodulation is allowed are based on the 
assumption that overmodulation occurs whenever the 
complex modulating signal exceeds the carrier level in 
either direction from the reference zero of the modulating 
signal; that is, both tails of the probability distribution 
have been included. This is not strictly correct in the usual 
case. Overmodulation certainly occurs when the carrier 
is reduced to zero but on the positive swings of the 
modulating signal, the point at which overmodulation 
occurs is not so clearly defined since it will depend in part 
on the saturation characteristics of the modulated 
amplifiers. Conceivably it is possible, and perhaps ad- 
visable in certain situations, to design the modulation 
stages to handle, with low distortion, positive swings of 
the signal greater than the unmodulated carrier level. 
Under these conditions overmodulation time as shown in 
Table I will be reduced nearly half for a particular value 
of M. Stated differently, M can be increased somewhat 
for each overmodulation time shown in the table. 

Finally, the overmodulation times considered were 
arbitrarily chosen and it is evident that M can be 
accurately determined for any desired degree of over- 
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modulation in excess of one per cent from the distributions 
of Fig. 2. The degree of overmodulation permissible in any 
situation will depend in part upon the noise inherent 
in the electrical equipment and surrounding environment. 
The best choice of overmodulation for a particular trans- 
mission system will be further dependent upon the 
number of subcarriers employed and the character of 
noise resulting from overmodulation. This essentially 
reduces to a determination of the resulting loss of infor- 
mation as a function of the degree of overmodulation, a 
topic for further study. 
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On the Response of a Certain Class of Systems 
to Random Inputs” 


JACK HEILFRON{ 


Summary—tThis paper deals with the connection between vector 
Markoff processes and the response of a lumped constant parameter 
linear system composed of a finite number of elements. It was 
known that if a Gaussian process which is one component of a 
vector Markoff process passes through such a system, the result is 
also Gaussian and may be considered as one component of a higher 
dimensional vector Markoff process. We show that the term Gauss- 
ian may be excluded in the above statement. The practical impor- 
tance of this result is that if one can determine the initial and 
transition probabilities of this vector Markoff process, one can also 
determine the complete statistical properties of the output of the 
system. This further implies that the determination of the properties 
of the output for the class of not necessarily Gaussian inputs men- 
tioned above is not as difficult as might be expected from the results 
available for just the first probability distribution for non-Gaussian 
inputs. 


INTRODUCTION 


URING the last decade there has been considerable 
1) interest on the part of engineers and physicists in 
problems dealing with random or stochastic proc- 

esses. One such problem concerns the determination of 


* Presented at the Western Electronic Show and Convention, 
Los Angeles, Calif., August 25-27, 1954. Work supported in part by 
the Office of Naval Research. 

+ Ramo-Wooldridge Corporation, Los Angeles, Calif. Formerly 
with the University of California, Los Angeles, Calif. 


the statistical properties of the output of a system, such 
as shown in Fig. 1, whose input is a known random 
process. In particular, the system is composed of a non- 
linear, no memory device commonly called a detector 
followed by a linear filter. This system could correspond, 
for example, to the final detector—video amplifier portion 
of a communication or radar receiver. Knowledge of the 
statistical properties (i.e., finite dimensional probabilities) 
of the output of such a receiver is required for any complete 
theory of signal detection in presence of noise.’ 


Detector Linear Filter 
v k(t, %) 


As far as available results are concerned, it is well 
known that if the input is a Gaussian process and the 
system is linear (i.e., the detector degenerates into a 
linear device), then the output is also Gaussian and its 
mean and correlation function can be readily computed. 
These two functions completely describe a Gaussian 


Fig. 1 


1R. C. Davis, ‘“Detectability of random signals in the presence of 
noise,” Trans. IRE, vol. IT 3, pp. 52-62; March, 1954. 
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process. For other situations, the problem is much more 
difficult. Kae and Siegert® have given a method for 
determining the first-order probability distribution of 
the output provided the input is Gaussian and the non- 
linear device is a square law detector. Siegert* and Fortet? 
independently derived equations for a certain conditional 


probability distribution of the output for the cases in ° 


which the input is one component of a vector Markoff 
process. As we shall show, this conditional probability 
distribution is all that is required to describe the output, 
i.e., to obtain all of its finite dimensional probabilities, in 
the event that the linear filter of Fig. 1 is a simple low- 
pass filter. 

Actually both Siegert and Fortet investigated function- 
als of the form 


To = | Rica Wile eMicae (1) 


) 


which does not correspond to the output of the system 
shown in Fig. 1. If the filter is a simple low-pass filter then 
its impulse response is 


kt) =" hem eames (2) 


and the system output is 


Oe / Beet ae de aaa Re Viena 


(3) 
Thus, 


16) Leone if Bee Vicia (4) 


is of the form suitable for the application of Siegert’s and 
Fortet’s results. For more general filters, the system output 
may not be so easily related to a functional of the proper 
form. 

We shall show just what probabilities are actually re- 
quired for the more general filter case by showing the 
connection between the output of the system and a certain 
vector Markoff process. Extensions of Siegert’s and 
Fortet’s results and also the Kac-Siegert method have 
been made which will yield, in principle, the required 
probabilities.° To avoid undue complications let us closely 
examine the simplest special case, then generalize. 


SPECIAL CASE 


Let the input to the system be a Markoff process and 
the linear device be a simple lowpass filter. The output 


2H. Cramer, ‘‘“Mathematical Methods of Statistics,’ Princeton 
University Press, Princeton, N. J.; 1951. 

3M. Kac and A. J. F. Siegert, “On the theory of noise in radio 
receivers with square law detectors,” Jour. Appl. Phys., vol. 18, 
pp. 383-397; 1947. 

+A. J. F. Siegert, “Passage of stationary processes through linear 
and nonlinear devices,” Trans. IRE, vol. IL 3, pp. 4-25; March, 
1954. 

>R. Fortet, ‘Additive functionals of a Markoff process,” 
Math., to be published. 

6 J. Heilfron, “On the Response of Linear Systems to Non- 
Gaussian Noise,’ Ph.D. Dissertation, Dept. Engrg., Univ. of Calif., 
Los Angeles, Calif.; 1954. 
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of the system is 


yi) =f pe? VEels)] ae. (5) 
[hick eee 
yt) =f Bee? Viata)l ds 
= energy, (6) 
where ia 
ats) =f pes Vf) de (7 
= y(t se 7 erat tae (8) 


The joint probability density’ of the output at the times 
t,, --- , €, defined as 


PAYr 5 ty 2° , Yn 5 th) 
is related to that of the increments, defined as 
PAZ: 5h, °°: Zn 5 th), 
by the equation 
DAUi jt 7) Ue Gila) —— eZ ee 
provided that [from (8)] 
Lie Oe oe (10) 


Thus, to determine the finite dimensional probabilities of 
the output, we may confine our attention to the corre- 
sponding ones for the increments. Also by considering the 
values of the input at times 0, 4, , --- , t, and the joint 
density function 


P(Xo My Zi y Uy 5 by ee te Rs ; Ne (11) 
we have 
Lie > linge e/a) 
-| of pcgriOs Zap per Oak oes 
Linry pauls) 1 bese UE eae (12) 


Thus we may also consider the joint density function of 
the input and the increments given by (11). Expanding 
this density function in terms of conditional probabilities 
gives 

P( ao OZ: » vy 3b J teloes Pe rrerers al) 
= P(%o ; O)p(Z, , t% 5 t | a 5,0) --- 
WZ, ) Ln } tn | Xo 10; Ly 7 Uy ; ty ) aia Zine 9 Un-1 } eae (13) 


By examining a typical term, namely the joint density 
function of Z(¢;) and x(t;) on the hypothesis that «(¢,) = 
a ,0 = 0, ++, mandZ() = Zpo7 = a1, «= newe-see 


‘To avoid mathematical details, we shall assume that all density 
functions mentioned exist. 
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that 


WZ; , x; 54; | Xo UDA aeons eno Pa an FGA) 


= p(Z, , x; Ae | Cray Ud), (14) 


because of the definition of the increments, (7), and the 
fact that the input is a Markoff process. Thus 


P(Xo KOK Z, 7 U4 ota ) “3° BAe shies oats) 


= p(X ; 0) i DDE estan Deere ats) (15) 
j=1 

which states that {Z(¢;), «x(t;)} is a two-dimensional 

Markoff process. It is of a degenerate type, since the 

transition probability 


WZ; , x; 


is independent of Z;_, . Since we have assumed a know- 
ledge of the input process, p(x ; 0) is known. Therefore, 
one need only determine the transition probabilities of 
the vector Markoff process to obtain all of its finite 
dimensional probabilities. These in turn give the same for 
just the increments by (12) and thence of the output by 
(9) and (10). 

The transition probability of the above mentioned 
vector Markoff process is the function that, in effect, both 
Siegert and Fortet consider (note comments in the Intro- 
duction). It can readily be shown that the process 
{y(t), x(t)} is also a Markoff process but its transition 
probabilities are more difficult to work with. 


ea | Mite sb as) (16) 


GENERAL SITUATION 


Let us now generalize to the case in which the input, 
which we denote by 2x'(é), is one component of a vector 
Markoff process 


X(t) a 7), res) x (O)) 

and the filter contains a finite number of lumped, constant 
value elements. In this event, the filter’s impulse response 
1S 


M 
He oak Aine Rp weet ie Sn (17) 
l=1 
Thus, the output is (see Fig. 2) 
wo = f Ee Wada 
0 
M t 
= if Be Vie) as 
l=1 (0) 
M 
= dy, (18) 
l=1 


Heilfron: Response of a Certain Class of Systems to Random Inputs 61 


where 


y(t) = it Be oe Vie) dr; Die, eee) 
0 

By considering all of the y‘(t)’s and the vector X(t) as 

was done for the special case before, we find that the 


M + N dimensional vector 


gD), oO, Oy eee) ee) 


is a Markoff process. Also, a linear transformation of a 
Markoff process results in a Markoff process so that 


fy) POP fy" Oya Oh ares (21) 


is also an M + WN dimensional Markoff process and the 
output is one component of it (namely the first as we 
have written it). Its initial probabilities are known so one 
need only determine its transition probabilities. Equations 
for these probabilities have been derived.® Actually it was 
more convenient to work with increments as in the special 
case considered here. 


te res ie 


x (t) 
k,(t-2) = 6, 07 “a (t-%) 
Ce oe eee 
Filter 
Fig. 2 
CONCLUSION 


We have shown that if the input to the system of Fig. 
1 is one component of a vector Markoff process and the 
linear filter contains a finite number of lumped, constant 
elements, then the output is also one component of a 
(higher dimensional) vector Markoff process. The dimen- 
sionality of the ‘‘output’’ Markoff process equals the sum 
of the dimensionality of the “input”? process and the 
number of natural frequencies of the filter. 

As a final remark, it is known*” that a Gaussian process 
with a rational spectral density (power spectrum) may 
be considered as one component of a vector Markoff 
process and thus is a suitable input. Also the output of the 
system we have considered is a suitable input for another 
system of the same type. 

8 J. L. Doob, “The elementary Gaussian processes,’’ Ann. Math. 
Stat. vol. 15, pp. 229-282; 1944. 


M. C. Wang and G. E. Uhlenbeck, ‘“‘On the theory of Brownian 
motion—II,”’ Rev. Mod. Phys., vol. 17, pp. 323-342; 1945. 
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Noise in Driven Systems- 


J. M. RICHARDSONT 


Summary—It is known that a direct relation exists between the 
noise in a system in equilibrium and transient drift toward equi- 
librium. It seems that a similar relation should exist for a system in a 
nonequilibrium stationary state. It is now necessary to distinguish 
between two types of transients; those produced by selecting those 
systems satisfying certain initial and those produced by actual 
physical perturbation. It is shown that a simple relation exists be- 
tween noise and the transients produced by selection and that no 
relation exists in the case of transients produced by perturbation. 
In the equilibrium case it is shown that the two types of transients, 
though still logically and operationally distinct, can be described by 
the same impedance operator. 


Il. INTRODUCTION 


ANY YEARS ago H. Nyquist’ derived a simple 
formula connecting the noise in system in 
equilibrium with the transient response of 

the system. It is our purpose here to see to what extent 
his results can be carried over to a system in a non- 
equilibrium stationary state. By the latter we mean a 
system is more or less strongly driven by some constant 
force (applied by a second system, which itself cannot be 
stationary) in such a way that is statistically (Le., 
macroscopically) stationary while not in equilibrium. 
We call such a system a driven system. A good example 
might be a resistor or a germanium diode connected to a 
constant voltage source. It is known that strongly driven 
devices exhibit noise differing in magnitude and quality 
from ordinary thermal noise. It would be useful and 
interesting if the noise in such systems could be related 
to the transient behavior. 

There are two distinct means for producing transients 
in a system. The first is by selecting from an ensemble 
those systems satisfying certain initial conditions. Or 
alternatively, by selecting the starting times of observa- 
tion on a given system by the same criterion. We call this 
type a transient produced by selection. It should be 
emphasized that here no physical action has been directed 
against the system. The observer has only made a decision 
about what systems to observe or alternatively when to 
start observing a given system. 

The second type of transient is produced by applying a 
transient force to the system. We call this type a transient 
produced by perturbation. 

We will show that a simple relation between noise and 
the transients exists when the latter are of the first type 
and that no such relation exists when they are of the 
second; and further that, when the system’s normal state 
is an equilibrium, one of the two types of transients can be 


* Presented at the Western Electronics Show and Convention, 
Los Angeles, Calif., August 25-27, 1954. 

+t Ramo-Wooldridge Corporation, Los Angeles 45, Calif. 

1H. Nyquist, Phys. Rev., vol. 32, pp. 110; 1928. 


described by the same impedance operator. It follows that 
noise is simply related to both types of transients. 


II. Norsk AND TRANSIENTS PRODUCED BY SELECTION 


Here we investigate the relation between noise as 
defined by the autocorrelation function or power spectrum 
and the average transient behavior following a selection 
process. The normal state of the system is stationary, but 
not equilibrium. 

For the sake of simplicity we shall treat the problem in 
the classical approximation. A quantum approach intro- 
duces interesting additional difficulties; however, these 
are not closely relevant to the questions that concern us 
here. Let our information concerning the driven system 
be given by an observable a, which is a given function of 
the microscopic state of the system, that is, a function 
of coordinates and momenta. As an example, we might 
consider the driven system to be a resistor and the observ- 
able a to be the electric current flowing through it. 

To formulate the equation of motion for a, we are 
obliged to consider the whole isolated system. This means 
that we must consider the driven system in conjunction 
with the system that maintains it in a nonequilibrium 
stationary state. Let us call the latter the de driving 
system. It is not possible for the combined system to be 
as a whole in a nonequilibrium stationary state. In order 
to maintain the driven system in a nonequilibrium station- 
ary state (at least for a finite interval of time) the de 
driving system cannot itself be in a stationary state. 
Consider a resistor as an example of a driven system and 
a battery as an example of a de driving system (with 
appropriate electrical connections, of course). In order 
to maintain a constant voltage across the resistor for a 
certain length of time, the battery will suffer internal 
chemical changes. In short, we must use in the equations 
of motion the Hamiltonian H for the combined system: 
driven system plus de driving system. Furthermore, the 
distribution function. describing the statistical situation 
cannot be stationary in the strict sense (except for the 
special case of equilibrium). However, it can in effect be 
stationary in the more limited sense that mean values of 
functions of the microscopic state of the driven systems 
are constant, or very nearly so, during a time interval 
of sufficiently long duration. 

Returning now to the formulation of the equation of 
motion for the observable a, the time rate of change of a 
is given by the equation 


da. zy, (2e OH dd aH) 
eS = H\ = = 
dies ° = ee > dq. Op, 9p, dq, 


S=1 


I 


La, (1) 
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in which the operator £ is an abbreviation for the operation 
[(( ), H]. The solution of the equation of motion can be 
expressed in the form 


Cy ane X% =a, (2) 


The subscript ¢ means that the quantity to which it is 
affixed is expressed in terms of the co-ordinates and 
momenta of the system a time interval ¢ earlier. The 
subscript 0 implies that the function is expressed in terms 
of the co-ordinates and momenta of the same instant. In 
accordance with usual conventions the absence of any 
subscript at all will be taken to mean the same thing. 

In the present treatment, we will approach statistical 
questions from the standpoint of a statistical ensemble 
rather than a time average standpoint. According to 
previous statements this ensemble can be stationary 
(when not in equilibrium) only in the limited sense that 
the driven system appears stationary. Let the carets (_ ) 
devote an average in such an ensemble. We find first that 
the mean value of the observable a, giving our information 
concerning the driven system, is constant in time: 


(a) = @) =a’. (3) 

The average value of a at time ¢, given that a = x at time 
0, is 

a(t; x) = (a,6(a — x))/{d(a — 2)). (4) 


The function 6(a@ — «) is the Dirac delta-function of 
a — x and it performs the task of selecting those members 
of the statistical ensemble that satisfy the initial con- 
dition a = 2. 
Defining 
Aa(t; x) = a(t; x) — a’, 


Ar =xz-— a’, (5) 


we obtain the following identities 
[a Aa(t; x)(é(a — x)) = 0, 
[ aw Ax(d(a — x)) = 0. (6) 


The first expresses the fact that an average over all the 
possible initial conditions is equivalent to no initial con- 
dition at all. We obtain by simple manipulations the 
further results 


[a IRA tpt ets (oes ao) eae Ay Ned 


/ FCC = Neen CO (7) 


in which 
Aa, SS Ol a (Oh (8) 


Let us pause to consider the meaning of Aa(¢; x) and 
various relations involving it. As stated in the introduc- 
tion, there are two distinct ways of producing a transient 
response in a system: by selection and by perturbation. 
The function Aa(t; 2) is the response produced by selec- 
tion, the basis of selection being a = «x at time 0. An 
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important point here is that nothing physical is done to 
any system. It is just a decision by the observer of which 
systems to observe’. If one wishes to assume a time average 
in preference to an ensemble average viewpoint one may 
make the following interpretation of Aa(t; x). Suppose 
now we have one system, not a statistical ensemble, 
and that we have a device which starts recording the 
time history of a every time a = x. The average of such 
recordings is a(t; x) = a° + Aa(t; x), where now ¢ is not 
the absolute time, but the time elapsed since the start of 
each recording. 

The first part of (7) establishes a connection between 
the transient produced by selection, Aa(é; x), and the 
statistical properties of the entire ensemble embodied in 
the autocorrelation function (Aa, Aa). The autocorrelation 
function describes the noise in @ exhibited by each system 
in the ensemble. The power spectrum is given in terms of 
the autocorrelation function by the well-known Wiener- 
Khinchin Theorem. 

A simple relation between noise and transient response 
can be obtained by expanding Aa(t; x) in powers of Ax 
and neglecting quadratic and higher order terms: 


Aa(t; z) = g(t)Ax + O((Az)’). (9) 


Insertion of this result into the first part of (7) yields with 
help of the second (7) the result 


(Aa, Aa) 


I 


/ dx Aa(t; x) Ax(6(a — 2x)) 


git) | dx (Ax)%5(e = 2) 
g(t)((Aa)?). 


Thus the shape of the autocorrelation function is given by 
g(t) describing the transient response produced by 
selection. 

Since 


(10) 


(Aa,Aa) = (AaAa_,) 


(Aa_,Aa), 
gt) = g(—-2). (11) 


It follows from (10) that g(é) is an even function of time. 
It must be pointed out that this is not the principle of 
microscopic reversibility. If we were dealing with a set of 
observables so that a would be regarded as a vector and 
g(t) as a matrix, the principle of microscopic reversibility 
would have to do with the symmetry properties of the 
matrix g. In particular, if the normal macroscopic state 
toward which transients tended were an equilibrium state 
(not merely a stationary state, in the limited sense) and 
if the observables were all odd or even, the matrix would 
be symmetrical. But in the present case of nonequilibrium 
stationarity, no symmetry properties would exist.” 

2 Since we are working in the classical approximation, the physical 
perturbation of a system by the process of observation can be arbi- 
trarily small. : 

’ Except, of course, for the matrix equivalent of the statement 


that the scalar g(t) is an even function of time: i.e., that the matrix 
g(—t) is equal to the transpose of the matrix g(+1). 
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Ill. Mopiriep Nyquist THEOREM 


It is desirable to transform the results of the previous 
section into a form more closely related to the usual 
formalism of circuit theory. To this end we must define 
an impedance operator Z(d/dt) giving g as the response 
to an impulse (6-function) input, even though no such 
input exists physically. We define such an impedance 
operator by the equation 

a(S) — t')g(t — t’)] = cot — ¢’). (12) 
The quantity ¢ is an arbitrary constant to be assigned any 
value we choose depending upon later circumstances. 
The quantity 1(¢ — ¢’) is unit step function defined by 


iN (Aue ia =k Ue ginbag eer 


=ilh, (itt (13) 


Thus, according to (12), 1(¢ — t’/)g(@ — U) is the response 
of the system to the input cé6(¢— t’). Multiplying the above 
response and input by Aw and using (9) we arrive at the 
alternative statement that 1(/)A a(t; x) is the response 
to the input cAxé6(¢). It should be emphasized again that 
these inputs are physically fictitious and are introduced 
in a formal way for the purpose of transforming the 
previous results into a form more closely related to 
conventional circuit theory. 
The impedance operator can be shown to be of the form 
ou(4) = ‘ — g(+0) + il du Kawe* © 


) at 


A(T \nnaat c= GAT). (14) 
in which K(uw) may be determined readily from (12) by 
the use of the Laplace transform. K(w) may be interpreted 
as the factor coupling the present rate of change of 
Aa(t; x) with the past value Aa(¢ — wu; x). If the system 
in question is a nearly perfect resistor in that its frequency 
response is nearly flat at high frequencies and absolutely 
flat at low frequencies the function K(w) will attenuate 
very rapidly with increasing w. 

We introduce now the concept of random driving force. 
This is a fictitious input imagined to ‘‘cause”’ the fluctua- 
tions of a about its mean value. We define the random 
force e, by the relation 


—— (4) a ; 


We must now find the autocorrelation function for this 
random force. By straightforward manipulations in- 
volving the use of (12) we obtain 


ak, . d a\; 
{e,e,,) = Ct i) = 24 \al4, ae.de.) 


(15) 


= {4 alo ace — vy((ae)’ 
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+ I(t! — tg(t! — )K(de)’) 


A | (Jac ai ee a(S Naw = » Jeaay 


{[ a(S4) + 2{%) Jace — en fone 


This is the modified Nyquist theorem in the time 
domain. It connects the autocorrelation function of 
e,, C(t — t) = (ee), with Z(—d/dt) + Z(d/dt), the 
resistive part of impedance expressed in the time domain. 

It is desirable to exhibit C(¢) in alternative forms. By 
introducing the explicit expression (14) for Z(d/dt) we 
obtain from (16) the result 


CW) = [—29'(+0) 6) + 1K + (—)K(—d]e"((Aa)’). 
(17) 

By taking the Fourier transform of (16) we obtain the 

modified Nyquist theorem in the frequency domain: 


(16) 


Cw) = [ He dtc ‘°'C(t) = 2 Re Z(iw)e((Aa)”) 


= tP(w). (18) 


Here P(w) is the power spectrum as conventionally defined 
with the frequency w expressed in radians per second. 

To show that (18) reduces to the usual statement of 
the Nyquist theorem, let the statistical ensemble be in 
equilibrium instead of merely stationary, and further, 
let the observable a be the electric current and let ¢ be the 
inductance or whatever is the coefficient of d/dé in the 
impedance giving the response of the current to a voltage 
input. We then obtain 

1 Z ; 
RO z Cw), = = Re Zw) kT, (19) 
where k is the Boltzmann constant and 7 is the absolute 
temperature. 


IV. Response PRoDUCED BY A PERTURBATION 


We wish now to investigate the response of the 
driven system to an actual physical perturbation. More 
specifically, we must consider the interaction between a 
transient driving system and the previous driven system 
combined with the de driving system. We consider a 
statistical ensemble in which the driven system combined 
with the de driving system is initially stationary in the 
sense described in Section II. The initial distribution of 
the transient driving system in the ensemble is arbitrary. 
At the initial instant the interaction with the transient 
driving system is “turned on.’’ We then calculate the 
mean value of a at subsequent times. It should be empha- 
sized that there is here no selection of a subensemble; 
i.e., the observer makes no discrimination—he observes 
whatever happens. 

Now let the subscript 1 refer to the combination of 
driven system and de driving system and let the sub- 
script 2 refer to the transient driving system. Let the 
Hamiltonian of the total system be 
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H = H, + H, + aa, , 


when H, depends only upon the state of the combined 
system 1 and is identical to H used in previous sections. 
Likewise, H, depends only upon the state of system 2. 
The interaction term a,a, is presumed to produce effects 


(20) 


sufficiently small to admit the use of first order time- 
dependent perturbation theory. The function a, is an 
observable depending only upon the state of system 2 
and may be regarded as the transient driving force. The 
time derivative of a, , dependent upon state of the 
combined system 1, is assumed equal to the observable 
a used previously; i.e., 


a, = a. (21) 


The operator £ for the total system can be divided into 
three parts: 


Ses ch Seis; (22) 
where 
L, = [( » H,], 
L2 aa [( ); A, ], 
Li2 = [( es QQ]. (23) 


We assume that the statistical ensemble is described 
initially by a distribution function P. In keeping with 
earlier remarks we assume that 


Poel Pow (24) 


where P; , referring to the combined systems 1, is a dis- 
tribution in which the driven system may be regarded as 
stationary although not necessarily in equilibrium. We 
will consider both cases of equilibrium and nonequi- 
librium. The distribution function P, , referring to 
system 2, is assumed to be completely arbitrary. We 
introduce for convenience a symbol § denoting the opera- 
tion of summing over the states of the total systems. We 
assume that the class of states summed over is such that 
S is factorable in the following way 


Sie SiS. (25) 
Our problem is now to calculate by first order time- 
dependent perturbation theory the quantity 
a(t) = $PiP, exp {t(£, + L. + Ly) }a 
= (exp {t(£; + L. + Li.) }a). 
It should be noted that since 
a® = (¢'*'a) = (a) = (au) 


a 
ee qy Pim = 0, 


I 


(26) 


the deviations from a’ are the same as the quantities 
themselves; e.g., Aa = a, Aa = a, etc. The result of first 
order perturbation theory is 


aph= i du G(t — u)a(w), (27) 
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where 
G(t) = —8,Py(e'**a)[a, , log P?] 
= —((e"*'a) [ay log Pi): , (28) 
and 
O30) = SPiP> exp {t(£, + £, + L342) Jay 
= (exp {é(. + £2 + £),)}e,). (29) 
Thus, G(¢ — w) may be regarded as giving the response 


to an impulse input at time wu. It must be noted that when 
P} is a nonequilibrium stationary (in the limited sense, of 
course) distribution function there is no relation (in- 
dependent of the particular form of the Hamiltonian) 
between G(t) derived here and g(¢) derived in Section II. 
Thus in case of a system whose normal state is stationary but 
not in equilibrium there is no relation between the response 
produced by selection and the response produced by pertur- 
bation. Because of this, there is also no relation between 
noise and the response produced by perturbation. 

We will now show that when the combined system 1 is 
in equilibrium, not merely stationary, G(¢) is proportional 
to g(t). The equilibrium requirement means that 


Pi = exp [(A — H,)/kT), (30) 


where k7’ is as before the product of the Boltzmann 
constant and the absolute temperature, and A is the 
Helmholtz free energy which plays the role of a normaliza- 
tion constant. With this form of P? we find 


[a, , log Pi] = —[e, , H,)/kT = —a,/KT = —a/kT..3)) 
Substituting this result into (28) and using (10) we obtain 
G(t) = ((e“*a)a),/kT 
g(t)(a")s /kT. (32) 


In deriving (82) we have, of course, used the trivial fact 
that Aw = a. It follows further that by choosing ¢ in 
(12) equal to ((a’),/kT)* we may describe the response 
G(t) by the same impedance operator Z(d)/dt used in 
Section III. We obtain 


AT) = GR) =o oe (33) 
By rewriting (27) in the form 
att) = i du [l(t — wG(t — w)Jas(u), (34) 


and operating by Z(/dt), we obtain 


(5 Jats) = / du 6(t — u)a.(u) = a(t). (35) 
0) 
Pursuing further the equilibrium case we obtain the 
Nyquist theorem in either of the following two forms by 
setting ¢ = ((a”),/KT) * 


C(t — tv) = ‘[2(—4) si a() Jac e inher (36) 


2 
EO) ~ Re Z(iw)kT . (37) 
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Design and Performance of Phase-Lock Circuits Capable 
of Near-Optimum Performance Over a Wide Range of 
Input Signal and Noise Levels~ 


R. JAFFE} AND E. RECHTINT 


INTRODUCTION 


HASE-LOCK LOOPS provide an efficient method 
Pree detection and tracking of narrow-band _ signals 

in the presence of wide-band noise. This paper 
explains how minimum-rms-error loops may be designed 
if the input-signal level, input-noise level, and a specifi- 
cation for transient performance are given. However, the 
system performance of such loops departs rapidly from 
the best obtainable performance if either the signal or the 
noise levels are different from the design levels, and if no 
compensating changes are made in the loop. A marked 
improvement results if the total input power is held 
constant, regardless of signal or noise levels. It will be 
demonstrated that a fixed-component loop preceded by a 
bandpass limiter yields near-optimum performance over 
a wide range of input signal and noise levels. The following 
topics will be discussed: 


1. An outline of the theoretical design of minimum- 
rms-error, phase-lock loops when input-signal level, 
input-noise level, and a specification for transient error 
are given. 


2. The effects of different input levels of signal and noise: 
a. On asystem having a fixed-component loop that 
is optimum only for an original set of levels. 
b. On a system in which loop components main- 
tain optimum performance when the new levels 
are given. 


3. Characteristics of a bandpass limiter. 


4. A comparison of the effect of different signal and 
noise levels: 

a. On a loop using a fixed filter preceded by an 
automatic-gain-control (AGC) system that 
holds the signal level constant. 

b. On a fixed-filter loop preceded by a bandpass 
limiter. 

ce. On a variable-filter loop continually adjusted 
to be optimum. 


5. Experimental verification of the fixed-component 
loop preceded by a bandpass limiter. 


* Presented at the Western Electronics Show and Convention, 
Los Angeles, Calif., August 25-27, 1954. This paper presents the 
results of one phase of research carried out at the Jet Propulsion 
Laboratory, California Institute of Technology, under Contract 
No. DA-04-495-Ord 18, sponsored by the Department of the Army, 
Ordnance Corps. 

+ Jet Propulsion Laboratory, California Institute of Technology, 
Pasadena, Calif. 


DESCRIPTION OF THE LOOP 


The elements of a typical phase-locked loop are shown 
in Fig. 1. Signal input to such a loop may be assumed to 
be 1/2A sin [wt + 0,(0)] where A is the rms signal ampli- 
tude, w is the signal-center radian frequency, and @,(¢) 
represents the information content of the signal. It is 
assumed that the noise input to the loop is narrowbanded 
about the signal-center frequency and has an essentially 
flat spectrum over the band; the noise input is represented 
by N(é) sin wt. Output of the voltage-controlled oscillator 
(VCO) is assumed to have an rms amplitude of C, a 
center frequency identical to that of the signal, and a 
phase equal to @,(¢). 


JE A Sin(wT +8, (1) + Nt) sin wT 2K AC Snleyi Bt) 


+/2 Ky CN(t) sin 6,(t) 


J2C cos(wT +6,) 


Fig. 1—Typical phase-lock loop. 


Briefly, the operating principles of such a loop are as 
follows: The multiplier beats the signal input and the 
VCO output together, giving a low-frequency output 
proportional to the sine of 6, — 6, . The loop filter accepts 
only this low-frequency term which is applied as a control 
voltage to the voltage-controlled oscillator, thereby 
forcing the VCO output phase 6, to be equal to the signal 
phase @, . 

It is possible to make a phase-locked loop such that the 
loop need have a bandwidth only large enough to pass the 
difference between the signal frequency and the VCO 
estimate of the signal frequency. Since this difference 
frequency has considerably less variation than the actual 
signal frequency, the loop does not need nearly as large 
a bandwidth as would be needed if the loop were merely 
a tuned circuit placed between the system input and 
output, which would have to pass all frequencies over 
which the signal was expected to vary. 

Since the bandwidth of the tracking loop is much smaller 
than that of a comparable nontracking filter, the amount 
of noise passed on to the output is proportionally smaller, 
and the loop accordingly picks up a greater resistance to 
interference. 
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LINEARIZATION OF THE Loop 


To obtain a mathematical description of the loop suit- 
able for subsequent analysis, it is desirable to approximate 
linearly the nonlinear operation of the actual loop.’ 

The low-frequency multiplier output is 

OKPAC sm Go be + N(t) V2 C sin 6,(t), 
where Kj, is the multiplier constant relating the multiplier 
voltage output both to the amplitude of its two inputs 
and to their phase difference in radians. Since the filter 
following the multiplier is low-pass, the high-frequency 
term in 2 wt is disregarded. Under the assumption that 
the phase error 6, — 6, is small and that the input noise 
is uncorrelated with sine 0, , 


sin (6; — 6.) = 0, — 4 
and 
va 
The multiplier output may therefore be represented 
approximately as 


KyC[A(A, ae: 6.) 4 N(2)]. 


N(é) sin 0, = 


The nonlinear operation of multiplication may now be 
linearized to subtraction, as shown in Fig. 2, together with 
associated changes of (1) making the phase input to the 
loop proportional to 6, + (N/A), (2) inserting an amplifier 
of gain A after the subtractor so that the voltage feeding 
the filter will be the same as in the nonlinearized con- 
figuration, and (3) representing the VCO as an integrator 
which relates its phase output to the integral of its input 
voltage by the VCO constant K, . The phases around the 
loop and the transfer functions of the loop elements have 
been expressed in terms of their Laplace Transforms 
rather than as functions of time. 


c[a(s) 4N{s)] yc [(s)— 6, (s)] +N(s) 


Fig. 2—Partial linearization of phase-lock loop. 


Finally, the various gain coefficients may be combined 
into a single amplifier of gain K, as shown in Fig. 3, where 
K is evaluated in both radians and degrees. The product 
AK is defined to be the loop gain. 


CRITERIA FOR FILTER DESIGN 
The loop filter is important in insuring good loop 
operation and should be designed so that it performs two 
distinct functions: 


1The linearization appears quite legitimate, based on experi- 
mental evidence. 
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1. It should minimize VCO phase-noise jitter due to 
noise interference. 

2. It should maintain, at a specified amount, transient 
error in the VCO phase due to specified changes in signal 
phase. 

Transient error is defined as the infinite-time integral of 
(6, — 6,)° and is caused solely by signal phase variations. 
Otherwise expressed, 


[ (aw - oor a. 


q-1 rad / sec 
k= [volts] x K,, [rad x volts }"'x K, [ee see | 
rl cyc/sec 
= S60IG [ volts] EK [ degrees xvolts | x Ky | 


Fig. 3—Complete linearization of phase-lock loop. 


Turory oF Fitter DeEsiqn 


If the rms phase jitter due to noise interference is 
denoted by cy , and if the transient error, as previously 
defined, is represented by H; , the design criterion may 
be stated as follows: 

oy + NE; = >” = minimum, 

where is an undetermined constant, a Lagrangian 
multiplier used in the standard calculus-of-variations 
procedure. Theoretically, when the filter form has been 
computed (Appendix I), the value of \ may be determined 
from the requirement that the transient error to a specified 
signal must equal a specified value. Practically, » is 
evaluated from the loop-bandwidth parameter By , which 
will be described in a subsequent paragraph. 

Filters may be designed for many forms of signal phase 
inputs. It has been found that filters designed to have 
zero steady-state error to signal frequency changes (zero 
velocity-error loops) are relatively easy to mechanize 
and work well for a variety of inputs. This form of filter 
will be the only one discussed in the body of the paper, 
although the results of similar analyses concerning loops 
designed to follow phase steps and frequency ramps are 
discussed in Appendix II. 

Derivation of the optimum filter is given in Appendix 
I. It is shown that such a filter for zero velocity-error 
loops has the transform 


re ae yauaas 
NG oe aa 
where 
5 (Aw) TAGs 
j= Sa 2A. ; 
s (N 0/ Ao) 
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and where K represents the loop constant defined in Fig. 
3, Aw is the signal radian-frequency step, No . the ex- 
pected rms noise input of bandwidth Af, and Ao is the 
expected rms signal strength. 

The output phase jitter and transient error of such an 
optimum loop, when the input noise and signal levels are 
those expected, are, respectively, 


2 3 L No 2 
Tis 5 wa IBS PEON) rad , 
(Aw)? _ 


2V2B 


It may be shown that B, is proportional to the expected 
loop bandwidth; B, is approximately equal to two times 
the loop noise bandwidth and three times the loop 3-db 
bandwidth. If the loop bandwidth is specified, B, may 
therefore be determined and \ may be evaluated from the 
definition of Bo . 

It is important to note that the parameter By, varies 
with the expected input signal-to-noise ratio and that the 
value of the optimum filter varies both with By and with 
the expected signal amplitude A, . 

This filter form is therefore optimum only for a par- 
ticular input-signal level and a particular input-noise 
level. If the actual levels differ from their expected values, 
the filter is no longer optimum. 


and 


E; = rad’ sec. 


PERFORMANCE OF Loops Havina LEVELS OTHER 
THan Desian LEVELS 


It may be shown, by integrating the actual loop transfer 
function, that the actual output phase jitter and transient 
error in a fixed-filter loop fed with signal and noise levels 
different from those for which it was designed are, re- 
spectively, 


asi N? |e 1] : 
OK = B( qs 21/2 rad (1) 
and 
Eo (Aa) : srad° sec, (2) 


2/2 B (A/Ad) 


where A is actual signal level, NV, actual noise level. 

It also may be shown that an optimum variable filter 
with values continually adjusted with slowly changing 
signal and noise levels would have the transfer function 


ides, (ecg 
es BLN uae Maan 


K As 3 


(3) 


and noise jitter and transient error, respectively, of 


2. 8B) Vy Aa 


ee 2Af 


(Aw) (N/A? r 
2 \/ 2 B3 (No/ Ao) We 


Thus far, a filter has been derived that is optimum for 


rad” 


and 


ad’. 


: 
aes 
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fixed input levels, and its phase jitter and transient error 
have been obtained when the input levels matched and 
mismatched the design levels. 

A variable filter has also been derived that was assumed 
to adjust itself to be optimum for different signal and 
noise inputs. Expressions for the phase Jitter and transient 
error in this loop have been obtained. 

The problem remaining is how to obtain optimum or 
near-optimum performance over a wide range of input 
signal and noise levels, yet achieve such performance 
using a filter than can easily be mechanized—instead of 
resorting to auxiliary servo loops that continually readjust 
the filter to keep it optimum. 

One possible solution is to use an AGC based on the 
signal only. The AGC voltage may be obtained if the 
signal is multiplied by a 90-degree, phase-shifted version 
of the VCO output; the inputs to the multiplier will then 
be in phase, and the de multiplier output will be pro- 
portional to the signal level and may be used for the AGC. 
The AGC method has several disadvantages: (1) It intro- 
duces additional components. (2) It introduces additional 
time constants which may cause two-loop oscillation. 
(3) It usually results in systems with less dynamic range. 


CHARACTERISTICS OF A BANDPASS LIMITER 


An alternate solution, which appeared promising, was 
to use a limiter preceding the loop. If the noise input to 
the system increased, the signal strength at the limiter 
output would decrease because of the limiter property of 
holding total output power constant. However, the noise 
bandwidth of the loop is directly dependent on the signal 
amplitude at the limiter output. Therefore, the increase 
in input noise, by forcing down the signal amplitude at 
the limiter output, would reduce the loop bandwidth and 
require that the phase jitter at the VCO output be a smaller 
percentage of the input noise. The system would therefore 
appear to be self-compensating. 

The type of limiter to be considered performs perfect 
snap-action limiting, 1.e., its output is assumed to be +1 
for inputs greater than 0 and —1 for inputs less than 0; 
such a representation closely approximates an actual 
limiter fed with input signal and noise levels much greater 
than its limiting level. The limiter is followed by a filter 
which restricts the output to the zone centered about the 
input frequency and excludes the harmonics generated in 
the limiting action. 

Davenport’ proved that signal-to-noise ratio is essen- 
tially preserved in passing through a bandpass limiter. 
This fact has been experimentally verified at the Jet 
Propulsion Laboratory. Youla of this laboratory then 
proved that the total power output of a limiter in a given 
zone is constant. Mathematically stated: 

N’ 4+ A? = LI? = N2 + A? 
and 


? W. B. Davenport, “Signal-to-noise ratios in band-pass limiters,’ 
Jour. of Appl. Phys., vol. 24, pp. 720-727; June, 1953. 


1955 
00 
50 + 
--* SIGNAL AGC 
--- LIMITER 
—— OPTIMUM 
= 10 
5 
7 1 ees Bea eee 
a 
z 
a 
5 ' 
cet [ls DES el ee eee 
0.5 
US -20 ° 20 40 0 
NOISE-TO-SIGNAL RATIO N/A (db) 
Fig. 4—Total phase error for various types of first-order loops. 
where 
N’ and A’ = limiter input noise and _ signal levels, 
respectively, 
N and A = limiter output noise and signal levels, 
respectively, 
L = total power output in a given zone 
(constant), and 
No, Ao = output levels of the limiter for which the 


loop was designed. 


These results may be combined to give the ratio both 
of actual signal level to expected signal level and of actual 
noise level to expected noise level at the limiter output: 


Gta GO - Heh 


This relationship may in turn be applied to the ex- 
pressions for the phase jitter and transient error [see (1) 
and (2)] in a fixed loop, giving the phase jitter and transient 
error in a fixed loop preceded by a limiter. 


CoMPARISON OF AGC, LimIrER, AND VARIABLE- 
PARAMETER OptimuM Loops 


It is now desirable to compare the behavior of the 
various loops discussed. The basis for comparison will be 
the relative phase jitter, transient error, and total error in 
the respective loops. 

Total rms error is defined as 


D= Vo2 + VE? rad, 


and was the quantity minimized by the Wiener calculus- 
of-variations approach used in designing the filter. 

Curves have been plotted for a loop using a fixed filter 
preceded by an AGC system that holds the signal level 
constant, for a fixed loop preceded by a limiter and for a 
variable-filter loop continually readjusted to be optimum. 
Expressions for loops designed to track input phase steps, 
frequency steps, and frequency ramps are derived in 
Appendix II and plotted in Figs. 4, 5, and 6. 

Other theoretical curves for a zero velocity-error loop 
will now be discussed. For these plots the various para- 
meters have been chosen to agree with the experimental 
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Fig. 5—Total phase error for various types of second-order loops. 
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Fig. 6—Total phase error for various types of third-order loops. 


and configuration to be described in a later section. 
Specifically, the input bandwidth of the noise was 900 
cycles per second, and the loop noise bandwidth was 124 
cycles per second (1.e., By = 25); the loop was designed for 
an input noise-to-signal ratio of unity. 


oNB CURVE: SIGNAL AGC 
o@Na CURVE: LIMITER 


o@Ny CURVE: OPTIMUM 


RMS NOISE ERROR (*) 


va 25 1 5 10 50 100 
NOISE - TO- SIGNAL POWER RATIO (N/A)* 


Fig. 7—Comparison of theoretical noise error in various loops. 


Fig. 7 shows the rms phase-noise jitter to be expected 
in the different loops. All the curves coincide at the design 
point of unity input noise-to-signal ratio but diverge at 
other input ratios. The phase-noise jitter in a loop em- 
ploying an optimum variable filter is not necessarily less 
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Fig. 8—Comparison of theoretical transient error in various loops. 
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Fig. 9—Comparison of theoretical total error in various loops. 
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Fig. 10—Total error in signal-AGC and limiter-loop percentage 
deviation from optimum. 
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than the jitter in the other loops because the optimum 
loop is continually minimizing the total error rather than 
only the phase jitter or the transient error. Optimum 
performance may require that a slight increase in phase- 
noise jitter be allowed in order to obtain a considerable 
decrease in transient error. 

Fig. 8 shows the transient error to be expected in the 
different loops. Since the transient error depends only on 
the loop gain in a fixed-filter loop tracking a given input 
frequency step, the AGC loop, which has constant input- 
signal amplitude, has a fixed loop gain and therefore a 
fixed amount of transient error. 

Fig. 9 shows the total theoretically predicted rms error. 
However, it is easier to see the significant features of the 
loops if a plot is made showing the percentage by which 
the total error in the AGC and limiter loops deviates from 
the total error in the optimum loop (Fig. 10.) It is seen 
that the loop preceded by a limiter has a total error not 
more than 15 per cent greater than optimum, whereas the 
AGC loop has a total error exceeding that of an optimum 
loop by 45 per cent. 


EXPERIMENTAL CONFIGURATION 


It has been shown that a limiter loop is theoretically 
superior to an AGC loop and is nearly as good as a loop 
continually adjusted to be optimum by auxilliary servos. 
It remains to be shown that the theoretically derived 
results for such a limiter loop are experimentally verifiable. 

Experimental results were obtained using equipment 
previously designed.’ The experimental configuration is 
shown in Fig. 11. The noise was bandpassed to a 900-cycle 


VOLTAGE - 
CONTROLLED 
OSCILLATOR 


Fig. 11—Experimental configuration for limiter loop. 


bandwidth centered at 5 kilocycles by a steep-sided filter 
and was then added to the 5-kilocycle signal. The sum 
was fed to a limiter consisting of a four-stage pentode 
amplifier having back-to-back silicon diodes across the 
input of each stage. The limiter output was passed thre hh 
a steep-sided, 7-kilocycle, low-pass filter which remc 
all but the first zone of the limiter output. The outpu 
this filter then served as the input to the phase-locked lc 
The loop itself consisted of (1) a multiplier using sili« 
diodes in a bridge circuit, (2) a multivibrator-type V\ 


5 Equipment used was designed by MacMillan of this laborato 
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Fig. 12—-Loop filter form. 


having a low-pass filter on its output, which passed only 
the fundamental component of the multivibrator square- 
wave output, and (3) a loop filter as shown in Fig. 12. 

The clean signal was compared with the VCO output 
in a phase meter, and the recorder output of the phase 
meter was connected to an oscillograph which was zeroed 
at the steady no-noise phase difference between the signal 
and the VCO. Any error in VCO tracking was then plotted 
directly on the oscillograph. 


EXPERIMENTAL RESULTS 


Before discussing the data obtained from this experi- 
ment, it is interesting to look at the spectrum of the 
noise present in various parts of the loop. Figs. 13(a), 13(b), 
and 13(c), are, respectively, photographs of the noise 
spectrum at input to the limiter, at output of the limiter, 
and at output of the low-pass filter following the limiter. 

The spectra are displayed on a logarithmic ordinate, 
with zero frequency at the right of the photograph. The 
input noise is centered at 5 kilocycles, with 900 cycles per 
second bandwidth. The first zone of the limiter output 
looks quite similar to the input noise except for the 
presence of some very low-level (considering the log- 
arithmic ordinate) background noise generated in the 
limiter; also, the limiter input noise has been spread out 
in a manner similar to that predicted theoretically.* The 
last photograph shows the effect of the 7-kilocycle, low- 
pass filter in attenuating the limiter background noise 
past 7 kilocycles, implying that the higher-order noise 
zones generated in the limiter are cut off. 

Experimental data on the noise error were obtained 
simply by adding a known amount of noise to the signal, 
and recording the VCO phase jitter on the recorder as 
shown in Figs. 14(a), 14(b), 14(c), for input noise-to-signal 
amplitude ratios of 1, 2, and 4, respectively. It is interesting 
to note how the frequency of the output noise jitter 
decreases as the input noise-to-signal amplitude ratio 


4 J. L. Lawson and G. E. Uhlenbeck, “Threshold Signals,” Radia- 
tion Laboratory Series, vol. 24:59, Mass. Inst. Tech., 1950. 
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Fig. 14—Experimental phase jitter in limiter loop. 


increases. This phenomenon implies that the loop band- 
width is decreasing with increasing input noise-to-signal 
ratio, a result theoretically predicted above. 
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lig, 15—I0xperimental transient error in limiter loop. 


Experimental data on transient error were more difficult 
to obtain. In order to determine only the transient per- 
formance of the system, it was necessary to disconnect the 
noise source and reduce the signal output of the limiter in 
accordance with the limiter relationship already discussed 
(4). The transient error of the loop was then obtained by 
introducing a 2.5 cycle per second frequency step into the 
VCO and recording the loop phase error on the oscillo- 
graph. The step could have been introduced into either 
the signal source or the VCO; the VCO was chosen for 
convenience. 

Figs. 15(a), 15(b), and 15(c) are recordings of the experi- 
mental transient error corresponding to input noise-to- 
signal ratios of 1, 2, and 3, respectively. The ordinate scale 
is 10 degrees for every heavy line. At the design level of 
unity signal-to-noise ratio, the Wiener theory requires 
about 5 per cent overshoot; as was expected, both the 
initial pip and the overshoot increase as the loop gain and 
bandwidth decrease. 

Finally, the experimental results were combined to plot 
in Fig. 16, and in Figs. 17 and 18 (opposite) the experi- 
mental vs the theoretical values of noise error, transient 
error, and total error, respectively, in a phase-locked 
loop preceded by a limiter and employing a fixed filter 
that satisfied the Wiener optimum criterion for a design 
level of unity signal-to-noise ratio. The results are com- 
mensurate. 


CONCLUSION 


It has therefore been theoretically proved that a phase- 
lock loop preceded by a bandpass limiter approximates, 
over a wide range of input signal and noise levels, the 
optimum performance obtainable only with a variable 
filter that is continually readjusted to be optimum by an 
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auxilliary servo system. It has been shown that, in ad- 
dition to facility of mechanization, the limiter loop 
approximates optimum behavior considerably better than 
does an AGC-controlled loop in which the AGC is based 
upon the signal. Finally, the theoretical derivations lead- 
ing to the expressions for error in the limiter loop have 
been experimentally verified and the effectiveness of a 
limiter phase-lock loop has been confirmed. 


APPENDIx I 


DeRIVATION OF OptimuM FILTER? 


Let Y(s) be the loop transfer function. By definition 


62(s) ae Y(s) 6,(s) (5) 
and 
é AKF(s) 
V9) = 5 AKFO. 6) 


The total noise power at the VCO output is 


y= | Y(s) |? Sy(s) ds, 
where ®,(s) is the spectral density of the input phase 
noise. Assuming this spectral density to be flat, Figs. 2 
and 3 show that 

ING 
A’2Af 


y(s) = by(0) = 


where Af is the input-noise bandwidth. 
The transient error, previously defined as 


[ (am = aor ae 


upon application of (5), may be written 


oe . | Y(s) — 1 |? | 6,(s) |? ds. 


E7= 
Since the optimum filter is to be designed so that ox 


is minimized under the constraint that HL; have a specified 
value, it is desired that 


D>? = oy + NE? = minimum. (7) 


The optimum filter is to be physically realizable, which 
for the purpose of this paper means that the loop transfer 
function has no poles in the right half of the s plane. 
Therefore, although the standard variational techniques 
are to be applied to minimize >’, the integral expression 
for >* must be set up in such a manner that it yields a 
realizable filter. 

Expansion of (7) gives 


> = = pe ds Y(s) Y(—s)®y(0) 
+ d’*[Y(s) — 11[¥(—s) — 1] | @x(8) |?. 3) 


5 This derivation follows the approach to the Wiener filt 


; ; vheory 
given in unpublished notes by Rechtin of this laboratory. 
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Fig. 16—Experimental vs theoretical noise error in limiter loop. 


Applying the standard variational procedure to Y(s), let 
VAs) = Sten (8), 


where en(s) is the variation to be minimized on the true 
optimum Y(s), and Y*(s) is the estimate of Y(s). 

When Y*(s) is split into factors having positive and 
negative poles and substituted into (8), 


SY () + en(s)] 
ee On ee  e 


+5 [av - Yo - 6 


-{1 — Y(—s) — en(—s)] | 6,(s) |’. 


Setting the variation of >’ to zero at ¢« equals zero 
completes the standard variational procedure. 


pee 
€=0 a 2m) ies gs (9) 


&= (Y() + en(9) 


-{(¥(—s)) (A? | (8) |? + Sy(O)) — r* | A(8) |?} 
+ oe ee ds n(—Ss) 
-{(YQ)Q? | (9)? | + Sv0)) — »* | a(s) 7} (9) 


In order to keep all terms in (9) split into factors having 
poles in either, but not both, the right or left half-plane, 
it is convenient to define 


d*] (8) |? + by(0) = | ¥(8) |) = HO) ¥(—8). (10) 
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Fig. 17—Experimental vs theoretical transient error for limiter loop. 


La CURVE : 


TOTAL.RMS ERROR (°) 


2 4 
NOISE-TO- SIGNAL AMPLITUDE RATIO N/A 
Fig. 18—Experimental vs theoretical total error in limiter loop. 


It is now necessary to substitute (10) into (9) in such a 
way that the latter will be satisfied for realizable Y(s), 
whereas the remainder of the integral will reduce to zero. 

This can be effected by noting that 


[© awe as =0 (11) 


if Z(s) and W(s) are algebraic polynomials having poles 
only in the same half of the s plane. 
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Substitution of (10) into (9) and keeping together terms 
having similar poles 


Ps ds n(s)¥(s)[Y(—s)¥(—s) — ” | 6(s) |?/¥)] 
+ n(—s)¥(—s)[Y(s) (8) — d* | As) |?/¥(—s)] = 0. (12) 
It is now advisable to split 
| @,(s) |? = | oer ES 
v6) Vol, Avert ey 


where the right-hand side of the equation is the partial 
fraction expansion of the left-hand side, having denomi- 
nator terms in +s and —s, i.e., having poles in the left 
and right halves of the s plane, respectively. 
Substitution of (13) into (12) gives 

[ ds n(s)¥(s) { ¥(—s)¥(—s) — ’[(| A(s) |?/¥(s)). 

+ (| A(s) |?/W(s))-]} 
+f  dsal-sy(—9 {YO — NI O19) P/H—9)), 


Oi(8) |" Ks) = 20: 


Because of (11), the terms having similar poles drop out, 
leaving 


ec 


[a r@v@tvi-a¥(—9 +X) 40) [/¥O)-3 


+f  asal-sy(-s) 


-{Y(s)¥(s) — (| (8) ?/W()4} = 0. = 4) 


These integrals are identical except that one has an 
integrand in +s and one, in —s. Therefore, the integrals 
are merely the negatives of each other, so that equating 
one of them to zero satisfies (14). 

Setting the second integral to zero yields 


? | | @(s) |? : 
Vs) | vs) Ip oD 
which is the optimum loop transfer function composed of 
realizable terms. Note that all terms have poles in the left 
half-plane, thereby satisfying the requirement of physical 
realizability. 

In this paper, (15) will be solved only for input frequency 
steps, although the results for other type inputs will be 
given. 

Assume the input to be a frequency step 


1G) 


A 
6,(s) x > 


where Aw is the magnitude of the step in radians per 


second. 
Therefore 


i (Aw)” 


March 
and 
ls) P = d* | a0) |? + Sy(0) 
= Mo + 00) = B® 5,0), 
where 
pre Mos ae 
oe y(0) 
Factoring 
| (8) |? = ¥)¥(—s) 
= (| 2 a as a ale = aes - “i 
gives . 
BG, [2 ‘ Mao + a fis 


Substitution of (17) into the optimum-filter equation 
gives 


d?s7( Aw)? 1 
} (s) aa 2 i 9 9 2 2 ? 
&,(0)[Bs + V2 Bos + 8] Ls?[B5 — 2Bys — s®] I+ 
(18) 
Separating the last term of (18) into partial fractions, 
1 _ 1/B) , V2/Be 
s[BS + V2 Bos + 87] s s 


c ss d 

Ba NV 2) $.— Be V2 

The coefficients ¢ and d are of no interest since the 
terms containing them have poles in the right half-plane 
and are not to be considered because of the definition of 
[er eeir(Ch ss 

The first two terms may be considered as having poles 
in the left half-plane, since they represent the limiting 
case of 1/(s — e) where « is a small number representing 
the reciprocal of the de gain. Since the pole approaches 
the origin from the left, in such a limiting process, the first 


two terms are included in the [ _ ].. terms. 
Therefore 
d*(Aw)’s” | 1 v2 | 
Ys = = 
° 6y(0) [Bi + V2 Bos + 3?] LBos” % os if 


which, with the aid of (16), reduces to 


Boe yo Be 


AGS) = a ; 
°) Bien OI eis: 


As a check, it is interesting to note that Y(s) is quasi- 
distortionless to a step-frequency change, i.e., 


error(s) = 6,(s) — 6,(s) = 6,(s)[1 — Y(s)] 


3? 


Bo + V2 Bs 3 


so that, if 6,(s) = Aw/(s’), representing a step-fre 
input, the infinite-time error is zero; i.e., the loop 
distortionless to a step-frequency change: 


a 6,(s) 


ney 
quasi- 
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error(é) = s X error(s) = 0 


error(s) = 


lim. 
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lim 


to 
Since the optimum loop transfer function is 


Bites \7 0 Bes 
Be Ae A/2 Busi-Lis? 


Vis) = 


and 
AKF(s) 
ea => — 
ue) s+ AKF%() 
It follows that * 
7 BS 2 By 
I (s) ix : ae - ) 


which was to be proved. 
The noise error and transient error in such an optimum 
loop may be calculated as follows: 


5 1 un ; 
oy = ee | Y(s) |? y(0) ds 

Ae “29 ca \ 12 = 2 
1 — Sean 1:0:(s)0\" [1 Y(s)]° ds. 


These integrals may be evaluated by involved contour 
integration and have been tabulated conveniently;° the 
evaluation of these integrals led to all explicit expressions 
for ox and E> given in this paper. 


APpprENDIXx II 


FILTERS FOR DIFFERENT ForMS OF SIGNAL INPUTS 


Filter forms for fixed-component loops, fixed-component 
loops preceded by bandpass limiters, and_ variable- 
component loops will be given in this appendix, together 
with their associated transient responses and phase-noise 
jitters, for two other forms of signal inputs besides those 
discussed in the text. Derivation of these filters are similar 
to the derivation in Appendix I and are not repeated. 


Loops designed to track phase steps will be called first-— 


order loops; loops designed to track frequency ramps will 
be called third-order loops. Second-order loops designed 
to track frequency steps are discussed in the text and will 
not be discussed in this Appendix. 


(1) First-order-loop—If the signal input to the loop is 
assumed to be a phase step, i.e., 


then the filter for the first-order, fixed-component loop 


has the transform 


B, 


K(s) = Fale 


8 “Servomechanisms,”’ Radiation Laboratory Series, No. 25, 
pp. 369-370, Mass. Inst. Tech.; 1947. 
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and has phase jitter and transient error, respectively, of 


ae aX) 
nD a(S a 


, (A@)? 


Loe 2B,(A/Ap) ’ 


and (19) 


where B, is a quantity similar to By in the previous dis- 
cussion of the loop for tracking frequency steps, i.e., 


ee UE / Daf. 


Nye 


Substitution of (4) into (19) gives a total error in a first- 
order loop preceded by a bandpass limiter of 


ot « NEM ENE 
he aye YS (3 A2af)/\A,/*\2 \az2af NA 
= a N? )l NE + (No/Ad)* , No . L+ Way] 

DORs L+(N/A)? © AGNI + (Wo/ Ab) 1 

It may be shown, using the same techniques employed 

in Appendix III for second-order loops, that the optimum 


variable-component filter for the first-order loop has the 
transform 


cba 
BS) = FEA Ne 
and that the total error in such a loop is 
Pi ee eae Bi\ Alte (NEE) pee) N/No 
2 at ae ae NENW) WAG A/C |b peel ea 


- HCN) 


Plots for the total error in first-order loops with (1) 
fixed components preceded by a signal-based AGC, (2) 
fixed components preceded by a bandpass limiter and (3) 
variable components are given in Fig. 4. Design param- 
eters are 


No 

Ay ; 

Bo a 10 

Af = 1 megacycle per second. 


The input-signal change was assumed to be a 1-radian 
phase step. 

For comparison purposes, the total érror in each of the 
three types of second-order loops is given in Fig. 5. The 
input-signal change was assumed to be a 6 radians per 


second (~ 1 cycle per second) frequency step. 


(2) Third-order-loop—If the signal input to the loop is 
assumed to be a frequency ramp, ie., 


then the third-order, fixed-component loop filter has’ the 
transform 
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aye B; + 2Bos + 2B,s° and that the total error in such a loop is 
ny sr a op-nee? =(9)(¥)"(Ma)'"( Be) 4, (aa 
Poi 0 Ih y — = 
and has phase jitter and transient error, respectively, of 7 PENS NA Ao 2Af 3 ae a Ae i 
2 
: = (Gao Te 
oD NAC) NON ey a) an WG ( N? y) etm 
and SON AOR Ay AD 
Aa)* 
Ey = Gani pe y— 1B’ (20) Plots for the total error in the three types of third-order 
ore Se i rae : loops are given in Fig. 6. The input-signal change was 
atl assumed to be a | radian per second’ frequency ramp. 
Bo = | aoe | v2a7 
No/ Ao 
ApprnpDIx III 
Substitution of (4) into (20) gives a total error in a 
third-order loop preceded by a band-pass limiter of OptimMuM-FILTER DESIGN 
>? = g2 + RP (#:)) N° i. + | The optimum filter is specified by 
a= = On Yip oe 29 r= = 
2/_A°2Af |L4(A/A,) — 1 it Bia 
i Ql Ne | (A/Ab) iz KAS ile) 
2/| Ac2Af_| [4(4/A,) — 1] where 
: De \(Aw) DAF 
; estat aay ESN Ae 


Sale 1 RST Eb on eee ee 
7 Aron | (ae B varies in accordance with the signal- and noise-input 
EN eA? ae 1| levels and may be expressed in terms of B, , the fixed-filter 
bandwidth parameter, as follows: 


(A/ A)" 1 one pi (A/a) | 
(N/N.) . + CAEP 1 [Lt Wo/ Ad? _ i | (NIN) 
LV 1+ (N/A)? 1 + (N/A)? Therefore 
It should be noted that the total error becomes infinite py Ald | 14/5 ee | 
when the noise has reduced the signal level to one-fourth F(s) = (N/No) (N/No) (21) 
of the design level. KAs 


It may be shown that the optimum third-order, variable- 


i conn wee ; 
Fee ccp heciticcmetonn giving transfer function for optimum variable filter (8). 


All results described in this paper as pertaining to 


By Ara | AS 2b ele second-order, variable-filter loops were obtained by 
F(s) = BICCNG NG | ae [RONG NS) Se NNO substituting (21) into the expression of Y(s) in (6) and 
ua As” > evaluating that which was required. 
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